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COMPOSITE ULTRASOUND IMAGING 
APPARATUS AND METHOD 

This invention was made with Government support 
under Contract DE-AC04-94AL85000 awarded by the U.S. 
Department of Energy. The Government has certain rights in 
the invention. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention (Technical Field) 

The invention relates to imaging and more particularly to 
an apparatus and method of creating composite two dimen- 
sional (2D) or three dimensional (3D) video presentations 
utilizing ultrasound image data. 

2. Background Art 

Creating 2D or 3D images from different image sources is 
taught in several patents such as U.S. Pat. No. 5,099,846, to 
Hardy. However, the image data utilized in these devices is 
limited to expensive image sources such as computed 
tomography (CT), nuclear magnetic resonance (NMR), 
X-ray, and other known imaging modalities. These image 
sources contain sharp or clear image data, therefore filtering 
or image manipulation is relatively easy to do. The use of 
ultrasound images in these systems is absent due to the 
complexity of defining internal targets' as opposed to mere 
surface location. 

Complete images, including skin and bone have been 
obtained using either X-ray computer-assisted tomography 
(CT) or magnetic resonance imaging (MRI) but these meth- 
ods have multiple disadvantages. Both of these procedures 
are costly and slow. Typical CT and MRI machines will cost 
several million dollars for. initial purchase and can cost 
hundreds of thousands of dollars more For specialized facili- 
ties that provide shielding of patients and personnel. 
Furthermore, x-ray CT uses ionizing radiation that has been 
determined harmful to humans. Most computer workstations 
for this technology are based on proprietary technology and 
are limited to their ability to interface with other data 
processing systems. Because of these problems neither CT 
nor MRI systems are practical for regular use. 

The present invention can be beneficial as a medical 
diagnostic tool, for example, for locating broken bones, 
certain targets or foreign objects in a body. Due to the small 
size and comparably inexpensive cost of an ultrasound 
system, the invention can be invaluable in the field or in a 
clinical setting where other more expensive imaging sources 
are not available. 

Another example for the use of 3D ultrasound images is 
in the manufacture of prosthesis'. To accommodate indi- 
vidual differences current practice is to custom make each 
prosthesis. This process is time consuming, costly and 
requires a highly skilled prosthetist. Although prostheses 
have been improved in quality over the years based on 
improvements in the manual techniques and in the use of 
new materials, the basic techniques remain largely the same 
as in previous generations. 

As patients walk and age, their residual limbs change 
shape due to atrophy of the soft tissue, as well as callousing, 
edema, scar tissue development, aging, and other complex 
processes. These changes alter the limb/socket interface and 
can result in pain or skin breakdown. The average amputee 
will need three to five new prostheses within the first five 
years after amputation because of changes in the residual 
limb that result in "socket failure". For each Dew prosthesis, 
the patient must undergo the entire measuring the fitting 



2 

process and the prosthetist must begin anew the design and 
fabrication processes. 

This situation is complicated by the fact that current 
design and fabrication procedures are more an artisan's craft 

s than a science. Construction of the patellar-tendon-bearing 
(PTB) type device, which is best suited to ambulation of 
below-the-knee (BK) amputees, involves techniques that are 
labor intensive and result in inconsistent products. Acquiring 
data about the shape of the limb and identifying weight- 

10 bearing and potentially sensitive areas are critical early steps 
in ensuring an effective fit. It is important to note that the 
weight-bearing and sensitive areas are in close proximity to 
the underlying bone structure. It is necessary to modify the 
socket shape to provide pressure relief directly over bony 

15 prominences because pressure in these areas will cause 
immediate skin breakdown. Additionally, it is necessary to 
modify the socket shape to support the patient in areas that 
can bear weight. These weight bearing areas are identified 
by their proximity to the underlying bone structure. The 

20 prosthetist uses his/her experience and knowledge of 
anatomy to approximate locating the bone structure. 

There are several disadvantages of this method. First, 
because the process is manual, the final fit depends entirely 
on the precision modeling of the leg. That leg model is 

25 generated in a step-by-step method that can create errors at 
each step. Second, the original model of the leg is not 
preserved during this process. Therefore, if at the end of 
fabrication, the prosthesis does not fit and cannot be 
adjusted, the entire process must be repeated. Finally, 

30 because the process is labor intensive and takes several days, 
the product is expensive. 

As an alternative method to manual socket fabrication, 
computer-aided design and manufacturing techniques are 
now being applied to the problem of designing prosthetic 

35 sockets. The data required at the initial stage of socket 
design involves converting measures of residual limb shape 
into a computer-readable format. Several researchers have 
discussed methods of gathering shape data including the use 
of silhouettes, as shown in "Silhouette Shape Sensor", 

40 Smith, et al., Bioengineering Centre Report., pp. 41-42 
(1986), digitizers, "Shoe Last Replication by Moire 
Contourography", Vickers, et al., Proceedings of the \th 
Rehabilitation Engineering Conference, Washington D.C., 
August 1981, Shadow Moire Contourography, "Moire Con- 

45 tourography and Computer Aided Replication of Human 
Anatomy", Duncan, et al., / Mech. Eng. 9, 1980; "Shape 
Sensing for Computer-Aided Below-Knee Prosthetic 
Design", Femie, et al., Prosth. Orth. Int., 9:12-16, 1985, 
light streak shape sensing, Proceedings of the '89 RESNA 

50 Annual Conference, New Orleans, La. June 1989; "Com- 
puterized Tomography as an Aid to Prosthetic Socket 
Design", Faulkner, et al., Rehab. R&D Prog. Rpt., 1:7-8, 
1987, computer assisted tomography, "An Ultrasound Shape 
Sensing Mechanism", Faulkner, et al., Presented at the 13//i 

55 Annual Meeting of the American Academy of Orthotics and 
Prosthetists, Tampa Fla. 1987, magnetic resonance imaging, 
"CAD/CAM System Applied to the Foot Shape for Pros- 
thetic Device", Oshima, et al., Proceedings of the RESNA 
8th Annual Conference, pp. 222-224, 1985, and linear 

60 potentiometers, "Socket Form Analysis of Computed 
Tomography Data", Faulkner, et al, Journal of Prosthetics 
and Orthotics, 1(3):154-164, 1989. Most of the work in this 
area has focused on utilizing surface shape data. 
The most commonly used alternative system of acquiring 

65 shape data employs a mechanical digitizer that measures the 
inside of a plaster mold of the patient's residual limb. It 
provides relatively low resolution because of its indirect 
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nature and is not effective in digitizing the distal end of the 
residual limb. Non-contact laser imagers can also provide 
shape information. Scanning is completed in just a few 
seconds and relatively high resolution data can be acquired. 
Mechanical digitizers and laser imagers only generate sur- 
face topology data. 

SUMMARY OF THE INVENTION 
(DISCLOSURE OF THE INVENTION) 

In accordance with the present invention there is provided 
an apparatus and method for providing two dimensional or 
three dimensional images from an ultrasound transducer. 
The preferred apparatus of the invention is an apparatus 
generating a video presentation of ultrasound images com- 
prising at least one ultrasound transducer, apparatus for 
acquiring a plurality of ultrasound images in scan incre- 
ments of an object, an apparatus for converting the acquired 
images into a selected format, an apparatus for storing the 
converted images and apparatus for registering the con- 
verted images to generate a composite image of the object. 

The preferred at least one ultrasound transducer com- 
prises a two dimensional array of ultrasound transducers. 

The preferred apparatus for acquiring a plurality of ultra- 
sound images comprises a apparatus for acquiring data from 
known orientations and positions. The orientations comprise 
angles from a home orientation. The preferred apparatus for 
acquiring data from known orientations and positions com- 
prises a feedback apparatus for optimizing a next scan. The 
preferred feedback apparatus comprises an apparatus for 
reorienting the at least one ultrasound transducer based on a 
previous scan's signal quality. 

The preferred apparatus for acquiring data from known 
orientations and positions comprises a filter apparatus for 
acquired data reduction. The preferred filter apparatus com- 
prises rank value filtering. 

The preferred apparatus for acquiring data from known 
orientations and positions comprises overlapping converted 
images. The overlapping converted images comprise verti- 
cal overlapping images. The overlapping converted images 
also can comprise horizontal overlapping images. The over- 
lapping converted images can also comprise overlapping 
images in any known angle between horizontal and vertical. 

The preferred apparatus for acquiring data from known 
orientations and positions comprises a position transducer. 
The preferred position transducer comprises a member 
selected from the group of encoders, potentiometers, 
LVDT's, resolvers, magnetic encoders, and inductosync 
apparatuses. The apparatus for generating a video presenta- 
tion with a position transducer further comprises a tachom- 

The preferred scan increments comprise surface normal 
optimization. The preferred scan increments comprise the 
composite image with an optimized intensity of predeter- 
mined anatomical features of the object. The preferred 
apparatus for registering comprises a transformation matrix 
apparatus for realignment of converted image data relative 
to home position orientation. 

The apparatus for registering can further comprises an 
apparatus for object motion compensation. The preferred 
apparatus for object motion compensation comprises corre- 
lating overlapping converted image data. The preferred 
apparatus for acquiring data from known orientations and 
positions comprises apparatus for determining coordinates 
for selected targets within the object from the converted 

A preferred method of generating a video presentation of 
ultrasound images comprises the steps of providing at least 
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one ultrasound transducer, acquiring a plurality of ultra- 
sound images in scan increments of an object, converting the 
acquired images into a selected format, storing the converted 
images and registering the converted images to generate a 

5 composite image of the object. The preferred step of pro- 
viding at least one ultrasound transducer comprises provid- 
ing a two dimensional array of ultrasound transducers. The 
preferred step of acquiring a plurality of ultrasound images 
comprises acquiring data from known orientations and posi- 

to lions. 

The alternative step of acquiring data from known orien- 
tations comprises acquiring angle data from a home orien- 
tation. The alternative step of acquiring data from known 
orientations and positions comprises optimizing a next scan 

1 5 with feedback from a prior scan. 

The preferred feedback comprises reorienting the at least 
one transducer based on a previous scan's signal quality. The 
preferred step of acquiring data from known orientations and 
positions comprises providing a filter for acquired data 

20 reduction. The preferred filter comprises rank value filtering. 
The preferred step of acquiring data from known orien- 
tations and positions comprises overlapping converted 
images. The preferred step of overlapping converted images 

2j comprise vertically overlapping images. The alternative step 
of overlapping converted images comprise horizontally 
overlapping images. Another alternative step of overlapping 
converted images comprise overlapping images in any 
known angle between horizontal and vertical . 

30 The preferred step of acquiring data from known orien- 
tations and positions comprises providing a position trans- 
ducer. The preferred step of providing a position transducer 
comprises providing a member selected from the group of 
encoders, potentiometers, LVDT's, resolvers, magnetic 

35 encoders, and inductosync apparatuses. The preferred 
method further comprises the step of providing a tachom- 
eter. 

The preferred step of acquiring a plurality of ultrasound 
images in scan increments comprise optimizing surface 
40 normal. The preferred step of acquiring a plurality of ultra- 
sound images in scan increments comprises optimizing the 
intensity of predetermined anatomical features of the object 
from the composite image. The preferred step of registering 
comprises realigning the converted image data relative to 
45 home position orientation with a transformation matrix. 
The preferred step of registering further comprises com- 
pensating for object motion. The preferred step of compen- 
sating for object motion comprises correlating overlapping 
converted image data. The preferred step of acquiring data 
50 from known orientations and positions comprises determin- 
ing coordinates for selected targets within the object from 
the converted images. 

A primary object of the present invention is to create two 
dimensional and three dimensional video presentations from 
55 a composite of ultrasound image data. 

Another object of the present invention is to provide 
detailed information regarding an amputees bone and 
muscle structure for prosthesis fabrication. 
6Q Yet another object of the present invention is to provide a 
diagnostic tool for imaging internal organs and structures 
using ultrasound image data. 

A primary advantage of the present invention is its low 
cost compared to other imaging modalities. 
65 Another advantage of the present invention is that it uses 
non-ionizing radiation. 

Yet another advantage of the invention is its portability. 
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Still another advantage of the present invention is the 
speed of creating the presentation compared to other imag- 
ing modalities. 

Another advantage of the present invention is its ability to 
clarify the composite images by motion compensation and s 
incident rank value filtering. 

Other objects, advantages and novel features, and further 
scope of applicability of the present invention will be set 
forth in part in the detailed description to follow, taken in 
conjunction with the accompanying drawings, and in part 10 
will become apparent to those skilled in the art upon 
examination of the following, or may be learned by practice 
of the invention. The objects and advantages of the invention 
may be realized and attained by means of the instrumen- 
talities and combinations particularly pointed out in the 15 
appended claims. 

BRIEF DESCRIPTION OF THE DRAWINGS. 

The accompanying drawings, which are incorporated into 20 
and form a part of the specification, illustrate several 
embodiments of the present invention and, together with the 
description, serve to explain the principles of the invention. 
The drawings are only for the purpose of illustrating a 
preferred embodiment of the invention and are not to be js 
construed as limiting the invention. In the drawings: 

FIG. 1 is a block diagram of the preferred embodiment of 
the invention; 

FIG. 2 illustrates the preferred hardware in accordance 
with the present invention; 30 

FIG. 3 depicts a typical horizontal reconstruction; 

FIG. 4 depicts a typical vertical scanning mode; 

FIG. 5 depicts a typical rank filtering horizontal recon- 
struction based on maximum signal echo; 3s 

FIG. 6 depicts a typical horizontal rank value filtering 
construction; 

FIG. 7 depicts a typical masked reconstruction; 

FIG. 8 is a comparison of before and after motion ^ 
compensated images; 

FIG. 9 graphically show the angular dependency of skin, 
bone and muscle layers; 

FIG. 10 shows one slice of a vertical reconstruction; 

FIG. 11 illustrates the preferred skin recognition process; « 

FIG. 12 depicts a typical 3D CAD mesh and visual 
rendering of a composite image; 

FIG. 13 is an example of segmented image with a user 
interface; Jo 

FIG. 14 is a block diagram of the preferred 2D horizontal 
scanning mode; 

FIG. 15 is a block diagram of the preferred 2D horizontal 
image acquisition structure; 

FIG. 16 is a block diagram of the preferred 2D horizontal ss 
image registration structure; 

FIG. 17 is a block diagram of the preferred 2D horizontal 
image motion compensation structure; 

FIG. 18 is a block diagram of the preferred 2D horizontal 6q 
image incident rank filtering process; 

FIG. 19 is a block diagram of the preferred 2D horizontal 
image composite image formation process; 

FIG. 20 is a block diagram of the preferred 3D vertical 
mode functional overview; 65 

FIG. 21 is a block diagram of the preferred 3D vertical 
mode vertical reconstruction process; 



FIG. 22 depicts the preferred ultrasonic diagnostic sys- 

FIG. 23 is a flow chart showing the preferred ultrasonic 
diagnostic system; 

FIG. 24 shows an ultrasonic sensor geometry modeled in 
computer software; 

FIG. 25 is a simulated leg geometry with 50 sensors and 
1 scan per sensor; 

FIG. 26 is a simulated circle geometry with 50 sensors 
and 1 scan per sensor, 

FIG. 27 is a simulated offset center circle geometry with 
50 sensors and 1 scan per sensor; 

FIG. 28 is an actual CT data scan; 

FIG. 29 is demonstrates the use of multiple scans (5) at 
each of 30 sensors; 

FIG. 30 is a three dimensional rendering of a leg; 

FIG. 31 is an ultrasound cross-section; 

FIG. 32 is a CT cross-section; 

FIG. 33 is a resulting edge from a CT scan; 

FIG. 34 is a side view of comparisons of three dimen- 
sional ultrasound and CT solid models; 

FIG. 35 is a top view of the models of FIG. 34; 

FIG. 36 is a side view of a registered ultrasound and CT 
volume; and 

FIG. 37 is a top view of the -volume of FIG. 36. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS (BEST MODES FOR 
CARRYING OUT THE INVENTION) 

The ultrasound apparatus and method of the present 
invention comprises hardware components and associated 
software for providing two dimensional or three dimensional 
images of an object. Ultrasound can be argued to be the 
logical choice as an imaging modality because it provides 
surface and subsurface information about a body part or 
other object to be studied. In addition, ultrasound is a 
non-ionizing form of radiation and currently, there are no 
known harmful affects from FDA approved ultrasound 
machines designed for medical diagnostic imaging. Ultra- 
sound machines are relatively low in cost (less than one 
hundred thousand dollars) as compared to CT or MRI. Most 
ultrasound machines provide resolution of images within a 
range that is acceptable for 3D imaging. For example, if the 
resolution in depth is 1.0 mm and the lateral resolution is 2.0 
mm, relative accuracies of each modality are comparable at 
1 .0 mm. Image frame rates vary depending on the number of 
focal zones and the dept of penetration from 30 frames per 
second (fps) to 7 fps. The slower frame rates reduce the 
speed of image acquisition. Even at 7 fps, however, a 
complete image set, for example of a leg, of 400 images can 
be acquired within 72 seconds. 

There arc two preferred embodiments discussed, one for 
prosthesis fabrication, and the other for use as a diagnostic 
tool. Although these are the only applications disclosed, the 
present invention can be utilized in other applications that 
presently use known scanning and imaging modalities. 

The preferred embodiment for prosthesis fabrication, of 
the invention is illustrated in FIG. 1. The 3D ultrasound 
imaging method and apparatus can be broken down into two 
separate sub-systems: image capturing and image process- 
ing. Image capturing comprises computer 10 that controls 
scanner 12 and inputs sync signal data 14, image data 16 and 
scanner position and orientation 18. The object to be 
scanned is placed in the scanning apparatus that can be 
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automatically or manually driven, such as the one shown in 
FIG. 2. Referring again to FIG. 1, in the automatic mode, 
scanner 12 is instructed via graphical user interface (GUI) 
20 within the computer 10 to scan the object with a pre- 
scribed level of precision. In the manual mode scanner 12 is 
instructed to accept force commands from the operator by 
grasping and moving the end of scanner 12. Thus scanner 12 
positions and orients transducer 22 to maximize image data 
signal 16 to ultrasound machine 24. Ultrasound machine 24 
provides images that arc captured by computer 10. Both 
position and orientation data 18 and images are stored 
simultaneously in computer 10 for reconstruction purposes. 
GUI 20 controls the position, orientation, and resolution of 
each scan and can also request historical data such as patient 
name, date and time of scan. 

The image processing subsystem takes the raw data 
acquired in the image capturing subsystem and forms a 
composite image of the anatomy that is scanned. Composite 
image is then processed using special image processing 
software 26 to enhance image quality. Motion compensation 
and filtering are performed prior to extraction of the skin and 
bone surface geometry. Image processing 26, motion com- 
pensation filtering are described herein. Data from skin and 
bone surface geometry can be used to fabricate custom worn 
devices such as prostheses. 

A visual presentation of the composite images is provided 
via display 28. 

FIG. 2 shows the preferred mechanical scanning system 
for prosthesis fabrication consisting of transducer holder 30, 
scanning tank 32, servo motor, and motor controller (not 
shown). Transducer holder 30 is a bracket that holds the 
transducer in a rigid fashion and maintains the position and 
orientation of the transducer in the scanning tank window. 
The scanning tank window is a flexible membrane that keeps 
water in tank 32 and allows ultrasound to penetrate without 
attenuation. Scanning tank 32 is filled with water for trans- 
mission of ultrasound for non-contact scanning of variable 
size legs. Non-contact scanning is essential for definition of 
"unloaded" soft tissue surfaces. 

The tank diameter (preferably 11.5 inches) can be used to 
accommodate the largest amputee leg size while optimizing 
the ultrasound, penetration range. Other sized tanks can be 
used for different body parts or targets for scanning. For the 
frequency of ultrasound, preferably 3.5 MHZ, the depth of 
penetration is limited to 22 centimeters or 8.7 inches. At this 
frequency, data can be scanned beyond the center of the 
tank. This frequency must be altered to accommodate larger 
diameter tanks. Thus, any object within the tank can be 
imaged without strict restrictions on its location. Servo 
motor (not shown) is controlled by position and velocity so 
that the proper number of frames can be captured around the 
leg. With position and velocity control, the servo-controlled 
scanner can be varied during image capturing. Thus, leg 
regions that are of more interest can be scanned with finer 
resolution of spacing then other regions. The position of the 
frame capture is recorded along with the image. This infor- 
mation allows the image processing package to reconstruct 
the images based on the raw data. 

The configuration of the three dimensional ultrasound 
scanning system allows for variations in the orientation of 
the ultrasound transducer. The two orientations are vertical 
(linear array oriented parallel to the long axis of the leg) and 
horizontal (linear array oriented transverse to the long axis 
of the leg). 

With the vertical orientation, scanning of the transducer 
circularly around the leg generates a volumetric data set. The 
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resolution of image spacing is measured in degrees. With 
variable scan rates, the bony regions of the leg are scanned 
with finer resolution that the soft tissue regions. Regardless 
of the resolution of image capture, there remains a space 
5 between captured data sets that is unaccounted for. In order 
to provide smooth transitioning between images, interpola- 
tion is used. 

With the transducer oriented horizontally, circular scan- 
ning of the transducer generates a planar image of the 

10 transverse cross-section. The data set is redundant with each 
point in the leg imaged multiple times at the different angles. 
The advantages provided by this method of scanning are 
many. Since each anatomical location within the leg is 
imaged multiple times, image processing algorithms can be 

is used to optimize the image based on the data. Consequently, 
the image with the most detail can be generated where each 
pixel within that image has been optimized for brightness 
and clarity. The resulting images from this technique can be 
used for diagnostic imaging that is comparable to the 

20 resolution of x-ray computed tomography. In fact, because 
ultrasound is better suited for soft tissue definition than is 
x-ray, ultrasound images demonstrate clearer tissue inter- 
faces. Volumetric image sets can be generated from hori- 
zontal scans by swiping the transducer vertically. 

25 The preferred ultrasound image acquisition system con- 
sists of Motorola 68030 based CPU, or the like, video frame 
grabber, motor controller, and an additional memory board, 
all housed in a VME bus based enclosure. A Sun-based 
graphical user interface (GUI), or the like, is used for 

30 command and control of the entire system. 

Utilizing the GUI, the operator can select the number of 
images to acquire from the ultrasound machine, choose the 
start and stop position of the scanner, as well as several other 
variables for correct operation of the system. Images can 

35 also be selected at constant or variable spaced intervals 
about the subject's leg. 
The VME CPU performs the following functions: 

a) Reads the command from the operator and calculates 
4Q the correct rotation rate for the scanner; 

b) Commands the scanner to start rotating; 

c) Continuously monitors the position of the scanner to 
decide when to grab the next ultrasound image; 

d) Grabs an image and stores it in memory; and 

45 e) On command from operator all images are transferred 
via Ethernet to host computer when operation is com- 
plete for post processing and image reconstruction. 
At the present time the output image data from most 
ultrasound machines is in analog format, and therefore must 
50 be digitized. It would be advantageous to obtain the data at 
the digital level, prior to image processing and output the 
video image. 

Image capturing and scan speed preferably are synchro- 
nized to a seven (7) Hertz image frame rate. This signal can 

55 be incorporated into the system. 

Ultrasound is most accurate when the beam is incident 
normal to the surface of the subject. As the angle between 
the transducer and the surface normal increases, image 
quality rapidly drops off due to reduced pixel intensity and 

60 inaccurate distance representation. 

The image quality of a leg is best when the incident angle 
is between +/- ten (10) degrees as the intensity values in the 
bone and skin images drop noticeably at larger angles. 
Image quality is best when ultrasound incident angles are 

65 between +/- ten (10), and inaccuracies in distance repre- 
sentations in the images occur with incident angles greater 
than +/- twenty (20). 
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The preferred prosthesis limb scanning apparatuses and 
methods acquire ultrasonic images in two modes: horizontal 
and vertical. In the horizontal scanning mode, individual 
frames are acquired at a predetermined angular increment 
about the limb, as depicted in FIG. 3. A cross-sectional 5 
reconstruction process is then applied which involves angu- 
lar rotations, motion compensation, and rank filtering, 
resulting in a highly defined planar cross section of the limb. 

In the vertical scanning mode, the transducer is oriented 
parallel to the vertical axis of the tank. Individual frames are 1Q 
acquired at a predetermined angular increment about the 
limb, as is seen in FIG. 4. A reconstruction process is then 
applied which involves angular rotations and polar 
interpolation, resulting in a volumetric reconstruction. The 
image and signal processing software developed for pro- 
cessing ultrasonic images can been integrated into the 15 
Khoros system, which is an algorithmic development and 
visualization environment. 

In the horizontal scanning mode, the linear transducer is 
oriented parallel to the bottom of the tank. In this mode, the 
x axis corresponds to the coordinate axis along the face of 20 
the transducer (or cross range), the y axis corresponds to the 
depth of penetration, or the direction along which sound 
travels iato the tank (or range), and e corresponds to the 
clockwise angle of the acquired frame with respect to the 
top, or front, of the limb. See FIG. 3. 25 

In this mode, a mechanical scanning system rotates the 
ultrasonic transducer about the limb. During the scanning 
process, individual ultrasonic frames are digitized along 
with the angle (©) of the acquired frame. The reconstruction 
process involves rotating the individual frames about the 30 
center of the trunk. In this reconstruction process individu- 
ally captured images are rotated with respect to the initial 
coordinate frame, and all frames are co-registered and 
combined to generate a composite image. It can be seen 
from the scanning geometry (FIG. 3) that an individual point 35 
within the limb will be visible to the transducer as the 
transducer sweeps through an angular spread. 

For prosthesis fitting, the points of interest are the surfaces 
of the tibia, fibula, and skin. Both bone and skin in ultra- 
sound data show up as relatively strong signal echoes. 40 
Exploiting the redundancy in the data, the maximum signal 
echo can be chosen from all those signal echoes correspond- 
ing to any given point within the limb. The resultant cross 
section reconstruction seen in FIG. 5 emphasizes the strong 
signal echoes of the skin and bone, but at the cost of also 45 
emphasizing any noise in the system, such as the tank noise 
around the limb. 

The noise signal echoes (tank reflections, bubbles etc.) are 
not coherent additions to the signal. Instead, the ultrasonic 
limb signal echoes replace noise signal echoes when present. 50 
The preferred technique under these circumstances is to use 
a rank value filter that employs a percentage ranking of the 
frames (instead of the average value over N frames) which 
preserves the desirable signal echoes (of the limb) while it 
eliminates the undesirable signal echoes (from the tank 55 
noise). Using the technique, FIG. 5 (the maximum signal 
echo) represents the 10096 ranking. In using this technique 
applied to the data set corresponding to that in FIG. 5 and 
choosing the 90% ranking, it produces the reconstruction 
shown in FIG. 6. 60 

Further, using the non-zero values of the reconstruction in 
FIG. 6 as a mask to the reconstruction shown in FIG.. 5 
produces the masked reconstruction shown in FIG. 7. This 
combination simplifies the task of recognizing the skin and 
bone signal echoes from within the reconstructed images 65 
because it maximizes the signal strength of the skin and bone 
surfaces while reducing noise. 



A horizontal scan that is generated between forty (40) and 
one hundred eighty (180) frames will typically require 
10-30 seconds to complete. Even during this short period of 
time patient motion can cause discontinuities in the recon- 
structed image. Any mechanical method to constrain the 
motion of the limb will distort the unloaded skin surface. 
Consequently, a small amount of motion during the scan 
should be expected. Any motion during the scan will mani- 
fest itself as a blur in the reconstructed image, thus adversely 
affecting the overall image. Depending on the angular incre- 
ment of successive frames of the acquisition (typically four 
(4) degrees), the majority of the information results from the 
same geometry within the limb. Because of this redundant 
information from frame to frame the data are highly corre- 
lated. Any motion of the limb between the acquisition of two 
frames will result in a shift of the correlation peak of the two 
frames with respect to each other corresponding to the 
amount of relative motion. The compensation for such 
motion is shown in FIG. 8. 

In the motion-compensated reconstruction image (FIG. 
9), the error has been reduced but not eliminated. This is a 
result of the integrated residual error over the entire scan. In 



the n 



the n 



tial frequency information of the structure of the internals of 
the limb is present. 

At first order, the ultrasonic signal echoes of a limb ' 
typically exhibit either reflective (specular) or volumetric 
scattering characteristics. Any given tissue type (bone, skin, 
muscle layers, etc.) is usually dominated by one of the 
characteristic types. Specular signal echoes usually fall of! 
quickly with changes in angles of incidence as compared to 
scattering signal echoes. During an acquisition scan, a two 
dimensional angle of incidence (versus signal echo value) 
can be generated for each point within the reconstructed 
image. Then, the angular dependency of a given point within 
the limb can be determined. Then, a classifier can be 
designed to recognize the differences in tissue type based on 
their angular dependency. 

From within the reconstructed image in FIG. 9, three 
points can be selected which correspond to a skin signal 
echo, bone signal echo, and a muscle layer signal echo. The 
plots exhibit the angular dependencies of these three tissue 
types. Notice the specular nature of both the muscle and 
bone, while the skin is much more scattering in nature. 

In the vertical scanning mode, the transducer is oriented 
parallel to the vertical axis of the tank. In this mode, the z 
axis corresponds to the individual ultrasonic transducer 
number (or height, increasing going up the limb), the r axis 
corresponds to the radius from the center (plumb line) of the 
tank, and 9 corresponds to the clockwise angle of the 
acquired frame with respect to the top or front of the limb. 
See FIG. 4. 

A mechanical scanning system rotates the ultrasonic 
transducer about the limb. During this process, individual 
ultrasonic frames are digitized along with the angle © of the 
acquired frame. The reconstruction process involves rotating 
the individual frames about the center of the tank. This 
process produces a series of frames which represent the 
ultrasonic signal echoes of the limb as the transducer is 
rotated about the limb. By contrast to the horizontally 
oriented transducer, where redundant information exists 
from frame to frame, individual points within the limb are 
observed only once during the scan, and some parts will not 
ever be visible to the transducer. See FIG. 4. 

A bilinear interpolation scheme is employed to produce a 
volumetric reconstruction of the limb. The bilinear interpo- 
lation is carried out for a given z (height) value, which 
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corresponds to a cross sectional plane of the reconstructed 
limb. The acquired data are presented in polar coordinates, 
but it is desirable to perform the reconstruction in Cartesian 
coordinates. Consequently, a polar to Cartesian transforma- 
tion is performed on the acquired data. For each point in the 
reconstructed plane, the four "surrounding" points from the 
acquired data are determined. A bilinear interpolation is 
performed to determine the value of the reconstructed point. 
The result is a "stack" of reconstructed cross sections of the 
limb, one corresponding to each z (height) value. See FIG. 
10. 

An individual frame from a vertical acquisition is shown 
in FIG. 10. This image corresponds to a vertical slice 
through the limb. The skin is recognized by correlating set 
of gabor filters, oriented at -30, 0 and +30 degrees, with the 
individual frame and determining the maximum response to 
the set of filters. See FIG. 10. The size of the gabor kernels 
is determined by heuristic examination of ultrasonic skin 
signal echoes. The leading edge of the skin response can 
then determined from heuristic examination of responses to 
the gabor filters. See FIG. 11. 

The skin locations are recorded in the form r, z, 8. These 
sets of points give rise to an overall three dimensional (3D) 
description of the skin surface of the limb. This 3D descrip- 
tion can be entered into a CAD package for potential 
fabrication of a prosthesis, or any 3D rendering package. See 
FIG. 12. 

Image processing operations can be performed within 
Khoros, which provides an environment for image process- 
ing algorithm exploration with industry standard routines for 
basic image conversion, processing, and display. 
Application-specific routines can be integrated into the 
Khoros system. They may contain calls to standard Khoros 
library functions or user written library functions that add 
new functionality to the system. 

A Visual Programming Environment (VPE) is also pro- 
vided that allows Khoros users to direct' inputs through a 
series of these library functions and display tbe results. With 
Khoros, a new algorithm can be rapidly prototyped and 
handed off to image analysts for review. The VPE can also 
be used by the image analyst for production work. Nine 
ultrasound specific application routines can be utilized and 
integrated into the Khoros system. These routines include: 
gabor — a routine used to generate a kernel containing a 
gabor function. The kernel can have arbitrary orienta- 
tion and size. The Gabor program can generate botb 
even and odd gabor functions, 
ncorr — will, calculate the normalized correlation of a 
kernel with an input image. The resulting image will be 
the extent of the input image. Normalized correlation is 
only calculated for those positions in which the kernel 
fits entirely within the input image. Edge effects are not 
handled. 

us2viff — "Ultrasound to Visualization Image File Format 
(VIFF)" conversion — us2viff takes the raw individual 
ultrasound frames in conjunction with the acquisition 
header information (sensor orientation, acquisition 
angle, date, time, subject, machine characteristics) and 
produces a multiband VIFF image file. The individual 
frames are stored as bands in the multiband image. 

ush2dxs — "Ultrasound horizontal two dimensional cross 
section generation" — ush2dxs rotates the individual 
frames about the plumb line to the angle which corre- 
sponds to the actual acquisition; compensates motion 
between the individual frames; and determines the 
maximum response throughout the frames on a pixel by 
pixel basis to produce a composite cross sectional 
ultrasonic image. 



)6,521 

12 

ushmocomp — is used to compensate the individual ultra- 
sound frames for motion of the limbs between succes- 
sive frames. It exploits the highly correlated nature of 
successive frames and determines the amount of 

5 motion by the shift in the correlation peak. 

ushplumb — is a calibration verification tool which will 
recognize the signal echo of a plumbob within the 
ultrasound imagery to verify the proper orientation of 
the sensor with respect to the center of the tank. 

jo Statistics are generated for both leading edge and 
maximum point signal echoes of the plumbob. 
ushrank — "Ultrasound horizontal rank value cross sec- 
tions generation" — routes the individual frames about 
the plumb line to the angle which corresponds to the 

15 actual acquisition; motion compensates motion; and 
ranks, across all frames, the individual pixel values to 
produce a set of ranked cross sections. These cross 
sections can be chosen to filter out incoherent noise, or 
other undesirable signal echoes. 

20 usvSdxs — "Ultrasound vertical three dimensional volu- 
metric reconstruction" — rotates (about the plumb line) 
the individual frames to the angle which corresponds to 
the actual acquisition; performs polar to Cartesian 
coordinate system transformation, and then executes a 

25 bilinear interpolation scheme to produce a series of two 
dimensional cross sections. The resulting "stack" of 
two dimensional cross sections is then assembled to 
produce an overall volumetric reconstruction of the 

30 usvskin — uses the individual vertically scanned frames to 
create a 3D description of the skin surface. The skin is 
recognized by applying a set of gabor filters to the 
individual frames and determining the maximum 
response to the set of filters. The skin location or 
35 leading edge of the skin response is saved in the form 
r, z, &). These sets of points give rise to an overall 3D 
description of the skin surface of the limb. 
Skin and bone surface segmentation can be performed 
from vertical scans with a flexible, interactive program that 
40 displays horizontal cut images made from the vertical scans. 
The program works on one cut at a time. The user begins the 
segmentation by providing an initial approximation on one 
cut. An automated segmentation program then takes over to 
refine the initial approximation. The final result is a file of 
45 three dimensional points made up of all the cuts. An example 
of a segmented image with the user interface is shown in 
FIG. 13. 

The user provides initial approximations of discrete con- 
tour points that give a rough outline of the skin and two 

50 bones. The graphical user interface displays the image data 
and gives mouse-driven, or the like, controls for making 
these approximations. To approximate the outline of the 
skin, the user sets a gray-level threshold that separates the 
darker image background from the brighter pixels on the 

55 skin. A radial line is traced from the outer edge of the image 
toward the center. The first skin point is taken at the place 
where this line first crosses the threshold. Starting from this 
point, the program traces the remainder of the skin by 
looking for nearby above-threshold values. The two bone 

60 contours come from pre-stored, approximate outlines of 
typical bones consisting of about 20 points each. The user 
drags these contours to the approximately correct location 
on the image. The program allows the user to rotate and 
scale all the contours and to move individual points of the 

65 contours. 

These initial approximations serve as the starting point for 
the automatic segmentation program. This program refines 
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the discrete contour points to make them fall precisely on the 
bright ridges in the image. The automatic segmentation 
program uses active contours called "snakes". Snakes work 
by solving an energy minimization problem for each con- 
tour. The "energy" is a sum of the (negative) pixel brightness 
along the contour and the bending energy of the contour. The 
bending energy is proportional to a weighted sum of the first 
and second derivatives along the contour. The positions of 
the contour points are iteratively moved to minimize the 
total energy. The brightness term tends to move the points 
toward image maxima, while the bending terms tend to keep 
the contour smooth. Together, these terms make the contour 
track bright ridges in the image while remaining smooth 
through gaps and noise. 

Image processing in the horizontal scanning mode is 
depicted in FIG. 14. Individual ultrasonic frames are 
acquired at a predetermined angular increment about the 
scanned object (in this case a lower human limb). A cross- 
sectional reconstruction process is then applied which 
involves image acquisition 100, image registration 102, 
motion compensation 104, incident rank value filtering 106 
and composite image formation 108. 

The image acquisition process and structure as shown in 
FIG. 15, comprise a commercial off-the-shelf (COTS) ultra- 
sound machine 110. A linear ultrasonic transducer is ori- 
ented parallel to the bottom of the tank and transverse to the 
scanned object In this mode, the x axis corresponds to the 
coordinate axis along the face of the transducer (or cross 
range), the y axis corresponds to the depth of penetration, or 
the direction along which, sound travels into the tank (or 
range), and theta corresponds to the clockwise angle of the 
acquired frame with respect to the top, or front, of the 
scanned object. 

In this mode, a mechanical scanning system rotates the 
ultrasonic transducer about the scanned object. The ultra- 
sound machine produces raw analog imagery 112 (RS170, 
video) while the mechanical scanning system produces a 
digital signal 114 containing the clockwise angle of the 
acquired frame. A computer system configured with a video 
frame grabber 116 is then used to digitize the ultrasonic raw 
frames 120 and record the digital signal containing the angle 
of the acquired frame. The ultrasonic raw frames and angles 
of acquired frames are then moved to the system's on-line 
storage 118. 

Image registration is shown in FIG. 16. From on-line 
storage 118, ultrasonic raw digital frames 120, along with 
the transducer orientation and the center of rotation, are used 
in a image rotation operation 122. Using the center of 
rotation and the transducer orientation, the ultrasonic frames 
are rotated about the center of rotation to the angle at which 
they were acquired. The resultant frames are spatially reg- 
istered 124. 

The motion compensation process is shown in FIG. 17. 
During the image acquisition phase, any motion of the 
scanned object will manifest itself by a blurring in the 
reconstructed cross section. It is desirable to sense and 
compensate for such motion. Depending on the angular 
increment of successive spatially registered frames 124 of 
the acquisition, the majority of the information results from 
the same geometry within the scanned object. Because of the 
common information from frame to frame, the data are 
highly correlated. A frame by frame image correlation 126 
operation is applied resulting in correlation surfaces 128. 
For each surface, a peak value (or greatest value) is detected 
130. The amount the peak is shifted from the center of the 
correlation surface gives rise to a motion-induced error 
estimate 132 (translation error). These motion-induced error 
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estimates 132 are applied in the form of frame by frame 
translations 134 to produce motion compensated images 
136. 

The incident rank value filtering process is shown in FIG. 

5 18. Within the motion compensated frames 136, an indi- 
vidual pixel in the acquired geometry (within the scanned 
object) will be repeated in a number of frames. Rank value 
filtering is applied by sorting the pixels within the scanned 
object with identical coordinates across all frames 138. The 

10 rank value images, sorted from minimum to maximum, are 
then stored 142. 

To produce the final composite image as shown in FIG. 
19, first the 90th percentile image is extracted 144 from the 
rank value images 140 (with 100th percentile being the 

15 greatest value). The 90th extracted image 144 is then thresh- 
old at the noise floor 146 of the ultrasound machine. This 
produces a binary image mask 148 with values of zero where 
the 90th percentile image was at or below the noise floor and 
one where the image was above the noise floor. An image is 

20 constructed from the pixels in the 100th percentile image 
that are in the 90th percentile mask image 150 (with value 
of one). The final composite image 152 that is formed is 
motion compensated and noise suppressed. 
For tissue classification as shown in FIG. 9, in the first 

25 order, the ultrasonic signal echoes of the human anatomy 
typically are described by either reflective (specular) or 
volumetric scattering characteristics. Any given tissue type 
such as bone, skin, blood vessels, muscle layers layers etc. 
is usually dominated by one of the characteristic types. 

30 Specular signal echoes usually diminish quickly with 
changes in angles of incidence when compared to scattering 
signal echoes. 

A typical 2D horizontal mode process involves the fol- 
lowing steps: 1) An angle of incidence, as compared to a 

35 signal echo value, is generated for each pixel within the 
scanned geometry; 2) The angular dependency of a given 
pixel within the scanned object is determined; and 3) a 
classifier recognizes the differences in tissue types based on 
their angular dependency. This is of particular importance 

40 when the above techniques are expanded to use in imaging 
abdominal cavities and other parts of the human anatomy. 

FIG. 20 depicts the preferred 3D-vertical mode functional 
overview. In the vertical scanning mode, individual ultra- 
sonic frames are acquired at a predetermined angular incre- 

45 ment about the scanned object (in this case a lower human 
limb). A 3D volumetric reconstruction process is then 
applied which involves image acquisition 200, image reg- 
istration 202 and vertical reconstruction 204. 

The image acquisition process and apparatus is similar to 

50 the 2D system in FIG. 15. However, the linear ultrasonic 
transducer is oriented perpendicular to the bottom of the 
tank and parallel to the scanned object. In this mode, the z 
axis corresponds to the individual ultrasonic transducer 
number, or height (increasing going up), the r axis corre- 

55 sponds to the radius from the center, or plumb line, of the 
tank, and theta corresponds to the clockwise angle of the 
acquired frame with respect to the top, or front, of the 
scanned object as observer from above the tank. 

Referring to FIG. 15, in this mode, a mechanical scanning 

60 system rotates the ultrasonic transducer about the scanned 
object. The ultrasound machine 110 produces raw analog 
imagery 112 (RS170, video) while the mechanical scanning 
system produces a digital signal containing the clockwise 
angle of the acquired frame. A computer system configured 

65 with a video frame grabber 116 is then used to digitize the 
ultrasonic raw frames and record the digital signal contain- 
ing the angle of the acquired frame. The ultrasonic raw 
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frames and angles of acquired frames are then moved to the 
system's on-line storage 118. 

Image registration in the 3D mode is also similar to the 2D 
system as shown in FIG. 16. From on-line storage 118, the 
ultrasonic raw digital frames 120, along with the transducer 
orientation and the center of rotation, are used in a image 
rotation operation 122. Using the center of rotation and the 
transducer orientation, the ultrasonic frames arc rotated 
about the center of rotation to the angle at which they were 
acquired. The resultant frames are spatially registered 124. 

The vertical reconstruction process is shown in FIG. 21. 
Polar spatially registered frames 205 are presented in cylid- 
rical coordinates (r,z,theta), but it is desirable to perform the 
reconstruction in Cartesian coordinates. A cylidrical to Car- 
tesian transformation 206 is performed on the polar regis- 
tered frames 205 producing a set of Cartesian planes, one 
corresponding for each individual z value in the acquired 
data. A bilinear interpolation scheme 208 is employed to 
produce a volumetric reconstruction 210 of the scanned 
object. For each point in the individual Cartesian planes, the 
four "surrounding" points from the acquired data are deter- 
mined. Abilinear interpolation is performed to determine the 
value of the reconstructed point. The result is a set of planes 
of reconstructed cross sections of the scanned object, one 
corresponding to each z (height) value. 

In operation skin recognition is performed by utilizing 
individual frames from a vertical acquisition which corre- 
sponds to a vertical slice through the limb. The skin is 
recognized by correlating a set of gabor filters against the 
individual frame. These filters are oriented -30,0 and +30 
degrees, and the maximum response from the three filters for 
each point in the frame is recorded. The leading edge of the 
skin response is then determined by the brightest pixel 
observed for a given z value. The z,r and theta for that skin 
point are recorded. The process is repeated for all values and 
all acquired frames giving rise to a complete three dimen- 
sional description of the surface of the limb. This 3D 
description can be entered into a CAD package for potential 
fabrication of a 3D model of the scanned object (limb). 

Once the image processing is completed, a digitized 
composite image can be utilized to fabricate a biomechani- 
cally correct prosthesis without reliance on an artisans 
experience. In the alternative, the digitized composite image 
can be displayed on a monitor, with the user manipulating 
the images for use as a diagnostic tool. 

Long term clinical research will benefit from the present 
invention by being able to track the atrophy of residual limbs 
over time. There has been no method of accurately measur- 
ing this phenomenon because suitable data has not been 
acquired. The soft tissue atrophies with time, but the bone 
structure remains much the same. Locating the bone struc- 
ture is necessary to track where the relative changes occur in 
the soft tissue. If the location of the bone structure can be 
determined using a non-contact ultrasound imager, then it 
will be possible to track patients over time using a low cost 
non-invasive procedure. 

In addition to prosthesis fabrication, the present invention 
can be used as a diagnostic tool for hospital, clinic or field 
use to replace the present three dimensional diagnostic 
systems. Present three dimensional imaging systems to not 
have motion compensation which would discourage use in 
the field and their expense and physical size is prohibitive 
for such uses. A portable diagnostic imaging in forward 
echelon combat areas would greatly benefit army medics to 
determine the extent of injury and focus care on those 
wounded that have the highest probability of survival during 
the first hour of injury. In addition, the system could be made 
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available to emergency medical technicians (EMTs) for 
civilian work. This information would provide high quality, 
high resolution, real time, 3-D ultrasound imaging necessary 
for diagnostics in the field that can be transmitted via 

s telemetry to physicians for a diagnosis for the harmful 
affects of ionizing radiation that accompany the use of 
x-rays. The three major contributing factors in successful 
high resolution ultrasound imaging are optimizing, image 
acquisition producing redundant data, image processing and 

10 image display. The ultrasound system comprises a tracking 
arm, transducer and coupling medium built into a hand-held 
device for remote imaging. 

The preferred ultrasonic diagnostic system is shown in 
FIG. 22. Similar hardware and software components as 

15 described in the prosthesis fabrication embodiment are uti- 
lized in this embodiment. A six axis robotic arm 250 is 
utilized with requisite motors and encoders for position and 
angle data. Force sensor 252 is attached to robotic arm 250 
to monitor and record force data. Ultrasound transducer 254 

20 is attached to coupling 256 as shown. Cup 256 with seal 
assembly 258 is affixed to ultrasound transducer 254. Water 
is pumped through cup 256 through water inlet 260 and out 
through outlet 262. 
Robotic arm 250 will help guide transducer 254 via 

25 passive or active path overlays to optimize image acquisition 
thereby maintaining perpendicularity to tissue, acoustic 
planes and controlling scan path trajectories. Data is pro- 
cessed for transmission with real time hardware. Telemetry 
transmission to the physician will allow them to further 

30 analyze the data or process the data for specific formatting 
such as two-dimensional cross-sections. 

The key elements providing a high quality, portable 
ultrasound system includes the following: coupling medium 
paradigm, which include water baths, standoffs, low flow 

35 water streams as well as other well known coupling medi- 
ums; electro-mechanical arm which allows control of the 
image acquisition path and includes position and orientation 
feedback, transducer steering based on computer generated 
or master input path overlays; support for the transducer 

40 cable and water supply lines and counterbalancing; velocity 
aberration correction based on redundant data; software for 
path planning of the transducer; real time hardware and 
software for image acquisition and image processing; image 
compression hardware and software; telemetry; and image 

45 display (2D or 3D, in real time). A flowchart depicting this 
system is shown in FIG. 23. Since ultrasound requires a 
coupling medium 300 in order to provide the transmission of 
an ultrasound signal, preferably water baths is used to act as 
the coupling agent. Other coupling agents such as stand off 

50 pads, gel pads or other known coupling agents in the art can 
be utilized. A low-flow, sterile, saline stream could be used 
a coupling medium as well as serving to provide sterile 
irrigation. Additionally, a water stream could be used to flow 
over non-penetrated skin surfaces without the need for a 

55 large source of water. However, flow rates, bubbles in the 
stream and fluid volume capacity must be considered. The 
most important component in obtaining high quality images 
is the optimization of image acquisition during scanning. A 
six degree of freedom mechanical arm 250 can be utilized to 

60 position and orient end mounted transducer on a standard 
ultrasound machine 254, and also to provide near perpen- 
dicular orientation and path control for redundant data 
gathering mechanical arm 250 will enable transducer posi- 
tion and orientation data to be available for reconstruction of 

65 ultrasound images. This is superior to any non-contact based 
device because it provides precise information without con- 
cerns of obstructive line of sight or metallic interference 
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issues that limit Don-contact devices. Mechanical arm 250 speckle techniques as well as image processing techniques 

can be made compact and can incorporate counterbalancing from digital signal processing, radar pattern recognition and 

devices allowing the transducer to be manipulating manually morphology. 

without difficulty. This will allow brakes and/or motors to be Phase-aberration correction techniques can be incorpo- 
incorporated to kinematically control the path, thereby, 5 rated in the system. These techniques are strongly coupled to 
optimizing the movement for near normal (perpendicular) the hardware design and require processing of the signal 
orientation during motion. Sensors such as reflected light from individual transducer elements. 'Methods for phase 
through the water path can be used upon perpendicularity for aberration correction can be applied to the linear, complex 
control adjustments (not shown). Alternatively, a master image that is available from some of the new ultrasound 
slave concept could be implemented to provide control of to machines such as the General Electric LOGIQ 700. 
transducer positioning to remote surgeons/physicians. The The present invention can provide redundant imagery 
master, controlled by the home base surgeon could be moved from multiple scanning geometries and form images exploit- 
in the direction desired. The slave controlled by the ing that redundancy. From the redundant imagery, rudimen- 
clinician, could be moved by the medic, with brakes or tary angular reflection and scattering characteristics of vari- 
motorsprovidingsteeringtooverlaythedesiredpath.com- 15 ous tissue types (skin, bone, muscle layers, etc.) may be 
puter generated overlays can also be used to optimize the extracted. Different tissue types will exhibit varying 
path and orientation of the transducer 254 over the body. characteristics, and classification of tissue based on the 
Mechanical arm 250 will also provide support for cables, the reflection and scattering properties is possible. Mathematical 
water supply to the transducer as would a cable tray. Robotic and statistical foundations to autonomously classify various 
simulation software, well known in the art, will be utilized 20 tissues types and techniques to display, manipulate and 
to optimize mechanical arms 250 range motion for applica- visualize the resulting two and three dimensional imagery 
tions to various human anatomy. Path finding algorithms can can be made. The results should enhance the medical 
also be employed that provided a model the scan path that expert's ability to provide high quality diagnostic informa- 
should be taken in order to optimize the image acquisition. tion to the clinician in the field. 

The traditional method of image compression for fast 25 High quality diagnostic images from a field portable 

transmission is to use a standard image compression algo- ultrasonic system should be provided to the rear echelon 

rithm such as JPEG or MPEG. Although these image com- hospital in near real time to enhance a patient's survival 

pression schemes could be employed to optimize the data during the first hour following injury, 

transmission, it would be beneficial to the physicians if The time latency associated with the image formation and 

"information triage" is performed prior to viewing. Infor- 30 processing techniques can be reduced by providing a system 

mation triage will take the form of image processing and solution combining Commercial Off-The-Shelf (COTS) 

segmentation to define the location of wound or shrapnel in hardware and system integration into a standard VME 

combination with localized blood flow from Doppler in chassis. The image forming and processing techniques are 

order to provide "improved" data to the physicians. By broken down into the basic image and signal processing 

performing this image processing prior to viewing, a data 35 operations required. The COTS hardware is then selected 

compression ratio of over 1000:1 could potentially be that maps these basic operations most efficiently. The basic 

accomplished for shrapnel and Doppler related data. operations include but are not limited to image capture, 

Remaining data compression will be accomplished as translation, rotation, spatial filtering, time series filtering, 

described below. convolution. The resultant system could be integrated into a 

Velocity inhomogeneities cause roughly two kinds of 40 Medical H for inclusion into an overall real time field 

distortion in ultrasound images. Small-scale inhomogene- portable ultrasonic system. 

ities cause blurred and locally distorted images, and this In the far forward battlefield scenario in which a medi- 
effect can be reduced with phase aberration correction. The cally equipped HMMMV would be use for initial medical 
other kind of velocity inhomogeneities are large-scale, and diagnosis of injuries suffered from conflicts in proximity to 
this is a potentially more serious problem than the small- 45 the Forward Line Of Troops (FLOT), it may be desirable to 
scale type, because they cannot be solved with phase aber- provide ultrasonic imagery via satellite link to rear echelon 
ration correction. Large-scale inhomogeneities occur when specialists up to hundreds of miles away (or even in the 
the tissues being imaged have significant differences in their continental United States). Critical is the ultrasonic image 
speeds of sound. These differences in velocity cause distor- compression technology that would allow ultrasonic imag- 
tion in the range coordinate (time axis) as well as distortion so ery from two and three dimensional composites produced to 
in the transverse coordinate due to bending or deflection of be compressed while preserving image quality and integrity, 
the rays representing the beam. Consider the example of two Because of the expense involve in high bandwidth SAT- 
homogeneous cubes, one contained in the other. If the COM links, it is desired to accomplish image transmission 
propagation velocities are different the image of the embed- over a Ti data link with bandwidth limitation of 1,500,000 
ded cube will be distorted and the distortion will be depen- 55 bits per second. This could provide a much broader area of 
dent on transducer/imaging geometry and aspect angle. coverage and flexibility of operation. 
Thus, large-scale inhomogeneities cause geometric distor- Ultrasonic and Synthetic Aperture Radar (SAR) imagery 
tions in the image that will affect the accuracy of a medical are very similar in nature and compression techniques that 
diagnosis. apply f° r SAR imagery should performs quite well for 
The problem of large-scale velocity inhomogeneities can 60 ultrasonic imagery. SAR is an army target recognition 
be solved with redundant imaging. By combining many system. The techniques derived for compression of ultra- 
different views of the tissue from multiple directions, one sonic imagery can also be applied to storage and recall of 
can explicitly detect geometric distortions. Redundant imag- imagery at rear echelon medical facilities or other central 
ing also has the effect of enhancing the image of smooth repositories for military medical records, 
(specular) objects or interfaces. Small features can be 65 The most cost-effective and reliable method of transmit- 
tracked from view to view, looking for size changes attrib- ting the data from the forward echelon to the rear is to 
utable to velocity inhomogeneities. This incorporates optical transmit the data in its raw format. This minimizes the 
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equipment and computational power required at the mobile 
site. In addition, all of the computational hardware is kept in 
a controlled environment — increasing the reliability of the 
overall system. The single computational unit can be used to 
process multiple input sources from multiple sites. Using s 
data compression and approximately 10 frames per second 
video, the raw data transmission rate can be reduced to 
between 1-4 Mbits/second. This data transmission rate can 
be implemented using standard off-the-shelf RF telemetry 
hardware. 10 

For far forward battlefield applications, portable, real- 
time hardware will be used to process data. However, in the 
rear echelon, access to an HPCC could be provided for 
processing power necessary for CPU-intensive activities 
such as real-time reconstruction of 3D images. These images li 
could then be viewed locally and/or sent back to the front 
lines via telemetry for viewing. 

EXAMPLES 

(Industrial Applicability) M 

The invention is further illustrated by the following 
non-limiting examples. 

Example I 

A first order model (simulation) of the ultrasonic imaging 25 
system was developed to evaluate the effects of refraction 
and scan geometry. The model was first developed in two 
dimensions and later extended to three dimensions. Geo- 
metrical optics were used to determine the image geometry. 
The acoustic beam is modeled as a Gaussian beam and the 30 
transducer is treated as a point receiving transducer. The 
limb being imaged is modeled as two materials, homoge- 
neous flesh and homogenous bone. 

Despite the simplicity, the model provides useful simu- 
lated images that illustrate some of the problems that are 35 
encountered in developing an ultrasonic imaging system. 
The model provides a useful estimate of image distortion 
and image brightness. The model includes effects due to 
refraction, acoustic impedance, attenuation, acoustic 
velocity, object geometry, and scan geometry. 40 

The following describes the elements of the computer 
model. The description will be developed in terms of the two 
dimensional model. This is adequate since reflection/ 
refraction are inherently planar calculations, that is, the 
calculations are carried out in the plane of incidence. Three 45 
dimensional aspects of the problem will be addressed as 
appropriate. 

The ultrasonic sensor geometry modeled in computer 
software of a typical leg is shown in FIG. 24. J() 

Ultrasonic transducer 22 is located at point A with its 
beam direction in the direction of the line AF. There are two 
types of scanning that are appropriate to model; a circular 
scan and a more general programmable scan. 

The simplest and most common type scan is a circular 55 
scan. This type of scan is determined by picking a center 
point F in the limb data coordinates. Transducer 22 is then 
located at radius AF with the beam oriented along the line 
AF. Transducer 22 is then rotated through 360 degrees about 
point F to generate the image. Scan increments are deter- 60 
mined by the resolution and image definition requirements. 
The point F is a variable in the program. 

The programmable scan is an option that allows the 
transducer position and orientation to be programmed to 
follow an arbitrary path. One scan geometry of interest is a 65 
linear scan where transducer 22 is moved along a line with 
the angle held constant. More general scans can be made up 
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from linear scans by summing linear scans with a discrete 
variation in the angle variable (parameter). Another scan 
geometry of interest is one in which transducer 22 is scanned 
over a range of angles at each position along a prescribed 

The angles of reflection and refraction are calculated 
using the equations 



sine. 



(2) 



where the subscripts refer to the incident and reflected rays 
and the c's are the acoustic velocities in the corresponding 
medium. Distances such as AB and BC arc calculated using 
an algorithm for computing the intersection of rays with the 
limb data skin and bone surfaces. The relative acoustic 
intensity of reflected and refracted rays are calculated using 
the equations 

r v , 1 2 (3) 
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In these equations, the numerical subscripts are ordered in 
the direction of the acoustic wave (ray) being considered and 
the p's are the corresponding material densities. 

The relative intensity of the acoustic beam is represented 
in angle as a Gaussian beam given by 



= exp[-(simj!/siii!j>o) ! ] 



(6) 



In equation, 7, X is the acoustic wavelength and d,, is the 
beam width at the beam waist (smallest width of beam). The 
beam width is usually taken to be the effective width of the 
transducer. However, the beam divergence angle can be 
empirically adjusted to account for scattering or receiver 
aperture width. 

Generally, acoustic beams decrease in intensity as they 
propagate due to absorption and scattering. In the model 
beam attenuation is incorporated as an exponential decrease 
by the equation 

/(*)-/(0)exp(- n) c] (8) 
where n is the effective absorption coefficient. In its present 
form, the model does not include a decrease in intensity due 
to the divergence of the beam. 

The relative image brightness determined by tracking the 
acoustic rays as they bounce off the various surfaces taking 
into account the loss of intensity due to reflection/ 
transmission and attenuation as appropriate. The computa- 
tion is illustrated for a few points in FIG. 24 by the 
following. 
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The relative image brightness of point B (relative to the 
transmitted intensity) in FIG. 24 is given by 



'The factor of 2 in the argument of p is due to the fact that 
the angle between the reflected ray and the ray returned to 
the transducer is twice the incident angle. 

The relative brightness of the return from point C is given 

by 



where the prime denotes the transmission in the opposite 
direction. One then continues in this manner, keeping track 15 
of the rays as they intersect the various surfaces. Attenuation 
is accounted for using equation 8 and the computed distance 
between consecutive reflection points. 

The image of point B is located a distance 



v R 



(11) 



from point A along line AF. V^, is the reference acoustic 
wave velocity. Similarly, the image point location of point C 
is given by the distance. 2 



again, along line AF. 3Q 

The two dimensional ultrasound simulation package is a 
C/C++ program that applies ultrasonic computation and 
image generation mathematics to mock ultrasound beams 
and predetermined tank and body part such as leg geom- 
etries. The calculations are applicable to a plane containing 
both the incident and scattered beam and a two dimensional 35 
leg model. A model is created of the acoustic properties of 
both the ultrasound beam and the materials it travels through 
with sufficient fidelity to predict the salient characteristics of 
the ultrasound image. The basic equations used in the 
simulation are given above. 40 

The algorithm requires input from leg and scan tub 
geometries along with the ultrasound beam parameters such 
as where the scans originate from, the direction they travel, 
and the strength of the beam. 

It also requires the acoustic properties of the materials in 45 
each region (i.e., water, muscle, bone, etc.). 

This is a collection of C and C++ programs with a 
Devguide based user interface. The output displays are done 
using Gnuplot windows. Beams are traced using recursive 
calls on reflected and transmitted components of rays as they SO 
intersect surfaces in the simulation. The return is always 
assumed along the incident beam path. 

A typical menu for the two dimensional simulator with 
details for each of the field/buttons are as follows: 

Coordinates of the center of the leg: This field allows the 55 
user to specify the X,Y coordinates of the center of the 
object being scanned. The coordinate parameters allow the 
object to be placed at various locations in the tank during the 
simulated scan. 

Nominal speed of sound (cm/sec): This field allows the 60 
user to declare the nominal speed of sound in water. 

Frequency in Hz: This field allows the user to specify the 
frequency of the simulated ultrasound beam. 

Lower limit of return signal: This field sets the lower limit 
of signal strength for return beams to be detectable. Initial 65 
transmitted power: This field sets the power of the initial 
beam transmission. 
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Focus beam radius (cm): This field sets the radius of the 
beam width at the beam waist. 

Max time for return signals: This field sets the maximum 
duration of time allowable for return signals to be detected. 

Number of sensors in scan: This field sets the number of 
discrete locations (or stops) from which scans are done. 
They are concentrically located around the "tub". 

Create scan path: This field initiates the creation and 
execution of the simulation using the specified parameters, 
generating a file containing the output results. 

Display plot: This field causes one of two types of display 
to appear. If simulated data, or CT data, have been chosen on 
"Data type," a Gnuplot window displays the results of the 
simulation run. If "three dimensional view" is selected, it 
causes a three dimensional grid display of the stored CT 
geometry but without scan data. Data type: 

Simulated data — selects simulated data for the scan — 
either a leg model or circles. 

CT data — selects leg geometries obtained from actual CT 
scans for the simulation. The user may choose any of 
the available CT cuts. 

Three Dimensional View — allows display of a three 
dimensional view of the "CT data." 

Model type: When "simulated data" is selected under 
"Data type", "Model Type" allows you to select either a leg 
model or circles for the scanned object. 

Number of sweep scans: This field selects the number of 
actual angular scans taken at each of the stops in the scan (at 
each of the discrete locations of the scanner). 

Sweep angle: The angle over which the "number of sweep 
scans" is uniformly distributed at each of the discrete scan 
locations. 

FIGS. 25-29 are examples of the simulation. FIG. 25 is an 
example of simulated leg geometry with 50 sensors, 1 scan 
per sensor using simulated leg geometries. FIG. 26 shows 50 
sensors, 1 scan per sensor for a circle. FIG. 27 is a simulated 
offset center circle. FIG. 28 is an actual CT scan. FIG. 29 
shows 5 scans at each of 30 sensors. 

The three dimensional model uses logic similar to the two 
dimensional model, but it extends into three dimensions. 
The differences include scan beams that have X,Y,Z com- 
ponents to both their origin and their direction. Points of 
intersection occur between three dimensional vectors and 
planes, as opposed to lines contained in the Z=0 plane, and, 
therefore, may have components in the Z direction. Finding 
the points of intersection is much more involved, and, 
consequently, the run time is much greater. Because the time 
constraints, this algorithm has only been implemented in 
batch mode with precalculated fixed scan parameters. The 
data input for the three dimensional mode contain: 

a file containing the parameters of the scan beams which 
includes the origin and direction (in three dimensional 
space) as well as the beam strength, created by a 
separate algorithm; and 

separate files for tank and leg geometries which define the 
three dimensional surfaces of the objects. 

The three dimensional simulation is a batch mode C 
program which uses three dimensional vector geometry to 
trace the beams with recursive calls on the reflected and 
transmitted components of rays that result from intersections 
with object surfaces. Rays are terminated when their power 
has dropped below a predetermined threshold, or are 
absorbed by the tank wall. Return signals are calculated at 
each intersection; again, assume a return along the incident 
beam path. Output is a file of three dimensional points. 

FIG. 30 shows the output points displayed as small black 
cubes with the three dimensional rendering of the skin and 
bone surfaces used in the scan. 
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Because the ultrasound machine provides a planar image bounding box. The next valid point on the edge is chosen as 
that represents a cross-sectional view of the object, an the one with the smallest vector angle with respect to the 
additional spatial dimension must be imparted to the trans- perpendicular. Continuing around the leg, angles are com- 
crucer in order to generate a three dimensional, volumetric puted to the points contained in the bounding box with 
image. The mechanical scanner provides the additional 5 respect to a perpendicular to the edge created by the last two 
scanning motion necessary to develop the third dimension of valid points. The next point chosen on the edge is the one 
the image. The scanner is designed based on the results of with the smallest angle. The process continues around the 
the computer mode] which concluded that a circular scan leg until the first point on the edge is contained within the 
around a BK limb is sufficient for providing skin and partial bounding box, indicating the completion of the edge See 
bone surface information. 10 FIG . 33 for an example of a resulting edge from a CT scan. 

Example II 7,1 ^ edging algorithm is used for edging the CT scans, 

including skin, tibia, and fibula, all of which must be done 

The 3D ultrasound imaging system was tested on 10 separately. The algorithm is also used for edging the ultra- 
unilateral below-the-knee amputees at the Health Science sound skin image, but cannot be used for the bones in the 
Center in San Antonio. Image data was acquired from both 15 ultrasound images. The ultrasound images are somewhat 
the sound limb and the residual limb. The imaging system coarse, and the bones are "incomplete" on the inside of the 
was operated in both volumetric and planar formats. An leg where the ultrasound has been reflected and attenuated 
x-ray CT scan was performed on each amputee for com- by the bone, shading the back portion of the bone. Therefore 
parison. Qualitative and quantative studies were performed the only valid bone data on the ultrasound scans is that near 
to compare CT and ultrasound data sets. Results of the test 20 the surface or skin of the leg. Instead of using the automated 
indicate beneficial use of ultrasound to generate databases edging algorithm, the bone images must be edged by "plac- 
for fabrication of prostheses at a lower cost and with better ing and re-sizing" templates of the bones on each cross- 
initial fit as compared to manually fabricated prostheses. In section. The marker affixed to the skin is visible at the upper 
addition, qualitative results indicate that planar images rep- right edge of the image in FIG. 31. 
resent substantial improvements over standard ultrasound « A graphical user interface (GUI) was designed which 
images and that they could serve as improved diagnostic aU 0WS me ^ to compare qua ii ta £ ve i y i mag 6 es obtained 
™ ages ' with the ultrasound platform to the CT images obtained. The 

In general, the human subjects placed their legs within the interface was written for use on a Silicon Graphics (SGI) 
water bath while holding onto the support railing. The machine, using "Forms" (SGI software for creating 
ultrasound gains were set for optimum image quality. 30 windows, buttons and other user interfaces), Graphics 
Images were stored on the computer for later retrieval and Language, and C programming 

manipulation. With the GUI, the user can load a series of images for a 

These ten subjects made up of 5 women and S men particular patient: ultrasound, CT, and edge points created 
provided databases for establishing the repeatability and 35 by the edging routine. See FIG. 31 for an example of an 
quality of the ultrasound scanning system. Each subject was image viewed using the GUI. The user may "browse" 
scanned using the vertical and horizontal oriented trans- through the series of ultrasound or CT cross-sections image 
ducer. Each leg on the subject was scanned to double the by image. In order to make a comparison, the user may, for 
effective data set. The system was fine-tuned based on the instance, display an ultrasound image at a particular position 
results of the scans. ^ on me leg . Next> tbe ^ can display the CT edge image> 

After each data set was obtained, the images were pro- generated from the corresponding CT scan taken at approxi- 
cessed through the visualization software used to generate a mately the same position on the patient's leg, overlaid in red 
composite image. For the men in the study, kg hair posed a on top of the ultrasound image. Features such as "click and 
problem for image acquisition. Leg hair tends to trap air drag", rotation, mirror, and scale are incorporated in the edge 
bubbles and tamed to blur the definition of the skin surface. 45 image display in order to properly line it up with the 
The volunteers opted to wear women's leg hosiery to keep corresponding ultrasound image. Once it has been posi- 
the leg hair close to the skin rather than shaving the leg. tioned for the "best fit", a visual comparison can be made of 
Movement, high reflection at the surface of the sock, and the proportions and positions of the features in the leg. 
poor centering were corrected for. Modifications in the scan Because of problems present with leg deformation in the 

procedure were made based on the "lessons learned" during 50 CT scans (due to parts of the leg being supported during 
the scans as wel as improvements to the visualization scanning), making comparisons between the ultrasound and - 
software. The result of this exercise was an improved image CT scans is difficult. Two dimensional comparisons are 
acquisition procedure and improved visualization software. greatly affected by any deformation since displaced tissue 

Algorithms were developed to "edge" accurately the two can cause cross-sections taken at the same position on the 
dimensional ultrasound cross-section images reconstructed 55 leg to look very different It was decided that three dimen- 
from the vertical scans and the two dimensional CT cross sional volumetric comparison methods would give more 
sections. See FIGS. 31 and 32 for examples of the ultrasound accurate results than two dimensional scan-by-scan com- 
and CT images. parisons. The theory behind this idea was that if any 

The outlines obtained from the ultrasound process can deformation occurred at a point on the leg, the tissue would 
then be used to produce three dimensional models of the leg. 60 be displaced a small distance away from its usual position. 
The images are first brought to a threshold to eliminate noise By examining the entire volume of the leg model, any 
in and around the leg cross section. The edging algorithm individual tissue displacements should not affect the overall 
starts from a "first" valid point, usually chosen by the user. volume calculation. 

From that point, the algorithm searches for the next point on Another GUI was created which displays three dimen- 
the edge within a bounding box of specified size. The initial 65 sional wire frame and solid polygon models generated from 
perpendicular to the surface at that point is assigned at 180. both ultrasound and CT images. See FIGS. 34 and 35 for an 
Vectors are defined from the first point to each point in the example of a comparison of ultrasound and CT three dimen- 
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siooal solid models. Edge points from the ultrasound and CT 
images for the specified patient are loaded by the applica- 
tion. The user can rotate and translate the models in order to 
facilitate viewing or to align the models for comparisons. 
The application contains a routine developed to iteratively 5 
orient the corresponding CT and ultrasound volumetric 
models by rotating and translating for the best geometric fit. 
Three dimensional point sets from the ultrasound data are 
registered with corresponding three dimensional point sets 
from CT scans. The two models can be registered by their 
entire volumes or only their intersecting length portion. If 10 
the ultrasound and CT models are separated initially, they 
should be registered based on their entire volumes since 
there is no intersection. If, however, the models have been 
approximately registered by the user beforehand, using the 
registration based on their intersection will refine the pose of 15 
the ultrasound point sets to find the best registration. See 
FIGS. 36 and 37 for examples of the ultrasound and CT 
models after registration. 

The point sets are registered based on Besl and McKay's 
(Besl & McKay, 1992) "iterative closest point" (ICP) algo- 20 
rithm. This algorithm searches for the minimum mean- 
squared distance between the two point sets. For each point 
in the moveable point set (ultrasound in this case), it finds 
the closest point in the stationary point set (in this case CT). 
The ICP then computes a transformation matrix to minimize 25 
the distances between the matched points. The moveable 
points are moved according to this computed transform. This 
process of matching and moving is repeated until the mean- 
squared distance cannot be reduced any further. 

Once the models are properly registered, a quantitative 30 
volumetric comparison can be made. In order to accurately 
compare the volumes of the ultrasound and CT three dimen- 
sional models, only the intersecting length portions of the 
two volumes are considered. This process is necessary 
because the CT volumes often represent a much longer 3S 
portion of the leg than the ultrasound data. The volumes of 
the two models are then calculated, and a percent difference 
value is given. 

The preceding examples can be repeated with similar 
success by substituting the generically or specifically 40 
described reactants and/or operating conditions of this 
invention for those used in the preceding examples. 

Although the invention has been described in detail with 
particular reference to these preferred embodiments, other 
embodiments can achieve the same results. Variations and 45 
modifications of the present invention will be obvious to 
those skilled in the art and it is intended to cover in the 
appended claims all such modifications and equivalents. The 
entire disclosures of all references, applications, patents, and 
publications cited above, and of the corresponding 50 
applications, are hereby incorporated by reference. 

What is claimed is: 

1. An apparatus for generating a video presentation of 
ultrasound images of an object, the apparatus comprising: 
a multielement ultrasound transducer that radiates ultra- 55 

sound energy in a first plane; 
means to move the transducer about the object in a second 

means for acquiring at least one plurality of partially 60 
redundant ultrasound images in tie first plane in scan 
increments of the object; 

means for converting said acquired images into a selected 

means for storing said converted images; and 55 
means for registering said converted images to generate a 
composite image of the object 



,521 

26 

2. The invention of claim 1 wherein said at least one 
ultrasound transducer comprises a two dimensional array of 
ultrasound transducers. 

3. The invention of claim 1 wherein said means for 
acquiring at least one plurality of partially redundant ultra- 
sound images in the first plane comprises a means for 
acquiring data from known orientations and positions. 

4. The invention of claim 3 wherein said orientations 
comprise angles from a home orientation. 

5. The invention of claim 3 wherein said means for 
acquiring data from known orientations and positions com- 
prises feedback means for optimizing a next scan. 

6. The invention of claim 5 wherein said feedback means 
comprises a means for reorienting said at least one ultra- 
sound transducer based on a previous scan's signal quality. 

7. The invention of claim 3 wherein said means for 
acquiring data from known orientations and positions com- 
prises a filter means for acquired data reduction. 

8. The invention of claim 7 wherein said filter means 
comprises rank value filtering. 

9. The invention of claim 3 wherein said means for 
acquiring data from known orientations and positions com- 
prises a position transducer. 

10. The invention of claim 9 wherein said position trans- 
ducer comprises a member selected from the group of 
encoders, potentiometers, LVDT'S, resolvers, magnetic 
encoders, and inductosync apparatuses. 

11. The invention of claim 10 further comprising a 
tachometer. 

12. The invention of claim 11 wherein said means for 
registering further comprises a means for object motion 
compensation. 

13. The invention of claim 3 wherein said means for 
acquiring data from known orientations and positions com- 
prises means for determining coordinates for selected targets 
within the object from said converted images. 

14. The invention of claim 1 wherein said partially 
redundant ultrasound images comprise vertical partially 
redundant images. 

15. The invention of claim 1 wherein said partially 
redundant images comprise horizontal partially redundant 
images. 

16. The invention of claim 1 wherein said partially 
redundant images comprise partially redundant images in 
any known angle between horizontal and vertical. 

17. The invention of claim 1 wherein said scan increments 
comprise surface normal optimization. 

18. The invention of claim 17 wherein said means for 
object motion compensation comprises correlating partially 
redundant image data. 

19. The invention of claim 1 wherein said scan increments 
comprise said composite image with an optimized intensity 
of predetermined anatomical features of the object. 

20. The invention of claim 1 wherein said means for 
registering comprises a transformation matrix means for 
realignment of converted image data relative to home posi- 
tion orientation. 

21. A method of generating a video presentation of 
ultrasound images of an object, the method comprising the 
steps of: 

a) providing a multielement ultrasound transducer radi- 
ating in a first plane; 
. b) moving the transducer about the object in a second 
plane; 

c) acquiring at least one plurality of partially redundant 
ultrasound images in a same plane in scan increments 
of the object; 
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d) converting said acquired images into a selected format; 32. The method of claim 23 wherein the step of acquiring 

e) storing said converted images; and data from known orientations and positions comprises deter- 

f) registering said converted images to generate a com- minin g coordinates for selected targets within the object 
posite image of the object. fro £ th f COn ^ d l m ?& S „ , . . r .. 

22. The method of claim 21 wherein the step of providing 5 „ iLT^T^r, T ^ ^ ° f aCqUinDg 
at least one ultrasound transducer comprises providing a tw5 31 ^ ° at J Am ^ y ? f P^y^undin unages corn- 
dimensional array of ultrasound transduced ' v^X™ 8 ^ ° f 

23. The method of claim 21 wherein the step of acquiring 34. ^ method of claim 21 wherein the ^ of ^ 

at least one plurality of ultrasound images comprises acquir- at l eas t one plurality of partially redundant images com- 
ing data from known orientations and positions. prises acquiring at least one plurality of partially redundant 

24. The method of claim 23 wherein the step of acquiring horizontal images. 

data from known orientations comprises acquiring angle 35. The method of claim 21 wherein the step of acquiring 

data from a home orientation. at least one plurality of partially redundant images com- 

25. The method of claim 23 wherein the step of acquiring prises acquiring at least one plurality of partially redundant 
data from known orientations and positions comprises opti- 15 images in any known angle between horizontal and vertical, 
mizing a next scan with feedback from a prior scan. 36. The method of claim 21 wherein the step of acquiring 

26. The method of claim 25 wherein the feedback com- at least one plurality of partially redundant ultrasound 
prises reorienting the at least one transducer based on a images in the first plane in scan increments comprise opti- 
previous scan's signal quality. mizing surface normal. 

27. The method of claim 23 wherein the step of acquiring 20 37. The method of claim 21 wherein the step of acquiring 
data from known orientations and positions comprises pro- at least one plurality of partially redundant ultrasound 
viding a filter for acquired data reduction. images in the first plane in scan increments comprises 

28. The method of claim 27 wherein the filter comprises optimizing the intensity of predetermined anatomical fea- 
rank value filtering. tures of the object from the composite image. 

29. The method of claim 23 wherein the step of acquiring 25 38. The method of claim 21 wherein the step of registering 
data from known orientations and positions comprises pro- comprises realigning the converted image data relative to 
viding a position transducer. home position orientation with a transformation matrix. 

30. The method of claim 29 wherein the step of providing 39. The method of claim 38 wherein the step of registering 
a position transducer comprises providing a member further comprises compensating for object motion, 
selected from the group of encoders, potentiometers, 30 40. The method of claim 39 wherein the step of compen- 
LVDT's, resolvers, magnetic encoders, and inductosync sating for object motion comprises correlating partially 
apparatuses. redundant image data. 

31. The method of claim 29 further comprising the step of 

providing a tachometer. * ♦ ♦ * , 
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[57] ABSTRACT 

An image combine apparatus for combining a plurality of 
images to generate a panoramic image. The image combine 
apparatus identifies an overlapping region of two inputted 
images and determines a boundary of the two images. The 
image combine apparatus then sets a tone correction area 
having a predetermined width such that the boundary of the 
two images is the center of the area, and performs tone 
correction within the area. The image combine apparatus 
performs linear tone correction in accordance with a dis- 
tance between a pixel and the boundary. In the neighbor of 
the boundary of the two images within the tone correction 
area, density of the image gradually changes, thus a com- 
bined image whose boundary of the two images is incon- 
spicuous is obtained. 

18 Claims, 20 Drawing Sheets 
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APPARATUS AND METHOD FOR 
COMBINING A PLURALITY OF IMAGES 

BACKGROUND OF THE INVENTION 
The present invention relates to an apparatus and a s 
method for combining images, and more particularly, to an 
image combining apparatus and method thereof for com- 
bining a plurality of images partially including an overlap- 
ping region having the same image, to generate one pan- 
oramic image having a wide angle of view. 10 

According to the conventional method of generating a 
panoramic image having a wide angle of view by combining 
a plurality of images partially having an overlapping region 
of the same image, two images are combined on a plane by 
geometrical transformation such as AfEn transformation or 15 
the like to coincide coordinates values of two corresponding 
points, which are extracted from the overlapping regions 
having the same image but having different coordinates 

However, in a case where conditions, particularly expo- 20 
sure conditions, of photographing the plurality of images are 
different for each image due to some factors of a photo- 
graphing subject, even if the two images are combined 
precisely with the corresponding points being coincided 
according to the conventional method, the boundary of the 25 
images would have a conspicuous line due to the difference 
in lightness of the images. 

The disadvantage of the conventional image combine 
technique is explained with reference to FIGS. 1, 2A, 2B and , n 
3. 

It is assumed herein that an image of a subject shown in 
FIG. 1 is picked up by an electronic still camera or the like, 
taking two frames (frame fl and frame £2) of photographs. 
Since the subject picked up by the frame fl has a dark 3S 
atmosphere as a whole, the camera corrects an exposure 
amount at the time of image pick-up such that the dark 
portion would be lighter. As a result, an image shown in FIG. 
2A is obtained. Since the subject picked up by the frame £2 
has a light atmosphere as a whole, the camera corrects an 40 
exposure amount at the time of image pick-up such that the 
light portion would be darker. As a result, an image shown 
in FIG. 2B is obtained. Accordingly, even if the two inputted 
images (images in FIGS. 1A and IB) are combined 
precisely, the combined image shown in FIG. 3 would have 45 
a conspicuous line due to the difference in lightness. 

SUMMARY OF THE INVENTION 



n has been made in consideration of 

> its object to provide an image 50 
combining apparatus and method capable of combining 
images in a manner such that the boundary of the images is 
inconspicuous even in a case where exposure conditions are 
different for each inputted image. 

Another object of the present invention is to provide an 55 
image combining apparatus and method which can generate 
a combined image whose boundary of the images is 
inconspicuous, by discriminating and identifying an over- 
lapping region having the same image in two inputted 
images, and correcting tones of either or both of the inputted 60 
images in accordance with difference in tone density 
between the two inputted images. 

Still another object of the present invention is to provide 
an image combining apparatus and method which can gen- 
erate a combined image whose boundary of the images is 65 
inconspicuous, by performing tone correction particularly in 
the neighbor of the boundary. 



Still another object of the present invention is to provide 
an image combining apparatus and method which can gen- 
erate a combined image whose boundary of the images is 
inconspicuous, by performing tone correction on pixels in 
the neighbor of the boundary in the overlapping region, in 
accordance with how far the pixel of interest is from the 
boundary. 

Still another object of the present invention is to provide 
an image combining apparatus and method which can gen- 
erate a combined image whose boundary of the images is 
inconspicuous, by removing erroneously recognized corre- 
sponding points from a set of corresponding points which 
have been detected to identify the overlapping region of the 
images, and by determining parameters of tone correction on 
the basis of the set of corresponding points from which the 
erroneous corresponding points are removed, thereby 
increasing precision of tone correction. 

According to a preferred embodiment of the present 
invention, since a coefficient of weighting is set in accor- 
dance with how far the pixel of interest is from the boundary 
of the images, it is possible to obtain a combined image on 
which smooth tone conversion has been performed. 

According to a preferred embodiment of the present 
invention, since the tone conversion is performed only 
within a predetermined area, the processing time can be 
reduced, moreover it is possible to obtain a combined image 
which is consistent with the image before conversion. 

According to a preferred embodiment of the present 
invention, since an area to be subjected to tone conversion 
is determined in accordance with the image in the overlap- 
ping region of the inputted images, tone correction and 
combine processing appropriate for the image are 
performed, thus possible to obtain a high-quality combined 

According to a preferred embodiment of the present 
invention, since an area to be subjected to tone conversion 
is determined in accordance with difference in average 
values of the images in the overlapping region of the 
inputted images, it is possible to obtain a combined image on 
which appropriate and smooth tone conversion have been 
performed. 

Other features and advantages of the present invention 
will be apparent from the following description taken in 
conjunction with the accompanying drawings, in which like 
reference characters designate the same or similar parts 
throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated in 
and constitute a part of the specification, illustrate embodi- 
ments of the invention, and together with the description, 
serve to explain the principles of the invention. 

FIG. 1 is a view showing image pick-up conditions of 
inputted images according to the conventional example and 
the present embodiment; 

FIGS. 2A and 2B snow examples of the input image; 

FIG. 3 shows an image combined in the conventional 
method; 

FIG. 4 is a block diagram showing an arrangement of an 
image combining apparatus according to the first embodi- 
ment of the present invention; 

FIG. 5 is a flowchart showing a process algorithm per- 
formed by a corresponding-point-extracting unit according 
to the first embodiment; 

FIG. 6A is a view illustrating a method of extracting a 
template utilized to extract corresponding points from input- 
ted images; 



10/15/2003, EAST Version: 1.04.0000 



FIG. 6B is a view illustrating a method of extracting a 
template utilized to extract corresponding points in a case 
where inputted images are arranged one on top of the other; 

FIGS. 7A and 7B are views illustrating a method of 
setting a searching objective area for extracting correspond- s 
ing points; 

FIG. 8 is a flowchart showing a process algorithm of tone 
correction; 

FIG. 9 is a graph showing characteristics of the relation- 
ship of sample data; 

FIG. 10 shows a table which stores values of conversion 
functions for a tone converting unit; 

FIG. 11 is a flowchart showing a process algorithm 
performed by an image combining unit; is 

FIG. 12 is an explanatory view of the image combining 
method; 

FIG. 13 is an explanatory view of the image combining 
method; 

FIG. 14 shows a combined image; 

FIG. 15 is a graph showing variance in density in the 
combined image shown in FIG. 14; 

FIG. 16 is an explanatory view for explaining extraction 
of sample data according to a modified embodiment of the 25 
first embodiment; 

FIG. 17 is a graph showing characteristics of tone con- 
version according to a second modified embodiment of the 
first embodiment; 

FIG. 18 is a graph showing characteristics of image 30 
combining process according to the second modified 
embodiment; 

FIG. 19 is a block diagram showing the arrangement of 
the image combining apparatus and the process flow of an 3J 
inputted image according to the second embodiment; 

FIG. 20 is an explanatory view showing a point Pi in an 
image a and a corresponding point Qi in an image b, 
according to the second embodiment; 

FIG. 21 is a flowchart showing the steps of tone correction 40 
processing performed by a correction-coefficient- 
determining unit and tone converting units according to the 
second embodiment; 

FIG. 22 is an explanatory view showing sample .data 
obtained to determine tone correction coefficients; « 

FIGS. 23A and 23B are graphs explaining the technique 
of tone correction according to the second embodiment; 

FIG. 24 shows a table utilized for converting pixel values 
in the image b to pixel values in the image a, according to 
the second embodiment; . 50 

FIGS. 25A-25C are explanatory views explaining the 
steps of removing erroneous corresponding components by 
a corresponding-point-selecting unit, according to the third 
embodiment; J5 

FIGS. 26 A and 26B are graphs for explaining the steps of 
determining tone correction coefficients by a correction- 
coefEcient-determining unit according to a third embodi- 



n of obtaining sample data 60 



Preferred embodiments of the present invention will be 65 
described in detail in accordance with the accompanying 
drawings. 



<First Embodiment 

FIG. 4 shows the arrangement of an image combining 
apparatus 1000 according to the first embodiment of the 

The combine apparatus 1000 may be a single apparatus, 
or may be built in a camera or the like. The apparatus serves 
to input two input images a and b, and to output a combined 
image c. 

Referring to FIG. 4, input images a and b are picked up 
by an electronic still camera or a video camera or the like in 
a manner such that both images partially include an over- 
lapping region having the same image. Even if the input 
images a and b are picked up with different exposure 
conditions such as those shown in FIGS. 2A and 2B (e.g. one 
image picked up in a dark condition, and the other in a light 
condition), the combine apparatus 1000 outputs a combined 
image c whose boundary of the images is inconspicuous. 

The combine apparatus 1000 includes a corresponding- 
point-extracting unit 1, parameter estimating unit 2, 
correction<oeffident-determining unit 3, two tone convert- 
ing units 4 and 5, and image combining unit 6. 

The corresponding-point -extracting unit 1 extracts corre- 
sponding points from the two input images a and b. Herein, 
corresponding points are each of the respective points in two 
images including an identical subject, which are picked up 
in separate image pick-up operation by the same light 
source. The corresponding points are outputted in the form 
of vectors. The corresponding-point-extracting unit 1 iden- 
tifies an overlapping region of images a and b. 

A parameter estimating unit 2 estimates parameters used 
for image conversion for the purpose of combining the two 
images, on the basis of the corresponding-points vectors 
extracted by the extracting unit 1. 

A correction-coeBBcient-determining unit 3 determines 
coefficients used for tone correction performed on each of 
the input images a and b. The correction coefficients are 
determined on the basis of image data in the overlapping 
region of the input images a and b. 

Tone converting units 4 and 5 respectively perform tone 
correction on the input images a and b such that lightness 
and tonality of every colors are equal in the overlapping 
region of the two images. 

An image combining unit 6 converts the input images a 
and b, whose lightness has been corrected, utilizing the 
conversion parameters and combines the two images into a 
single image. 

A control unit 7 controls the entire image combining 
apparatus. 

Corresponding Point Extraction by Unit 1 

Description will now be provided on the operation for 
generating a combined image c based on the input images a 
and b. Herein, description will be provided in a case where 
the input images a and b have N tones of density image data. 

The corresponding-point-extracting unit 1 extracts corre- 
sponding points from the input images a and b to identify an 
overlapping region. The process algorithm performed by the 
corresponding-point-extracting unit 1 is shown in FIG. 5. 
The algorithm shown in FIG. S is executed to perform 
template matching and extract corresponding points (i.e. 
overlapping region). 

In step SI, the corresponding-point-extracting unit 1 
determines a region which includes a template for perform- 
ing template matching (hereinafter referred to as a template 
region). Since the frames of the input images a and b are 
arbitrarily set, the overlapping region cannot be determined 
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in advance. Thus, the extracting unit 1 sets a predetermined 
region as a template region. 

The predetermined region can be determined in various 
ways. 

It is assumed in the first embodiment that the image of the 
frame fl is photographed first and the image of the frame f2 
is photographed next. Therefore, the input image a is 
arranged in the left side and the input image b, in the right 
side. In other words, an overlapping region of the two 
images should be the right portion of the input image a and 
the left portion of the input image b. Accordingly, a prede- 
termined region T is set as shown in FIG. 6A. The region T 
is located in the right portion of the image a. The horizontal 
length of the region T is 30% of the horizontal length of the 
image a, and a vertical length of the region T is 80% of the 
vertical length of the image a. Note that the region T is 
located in the center of the vertical length of the image a, 
with the right end thereof adjoining to the right end of the 
image a. 

Note that in a case where the input images a and b are 
arranged vertically, one on top of the other, the template 
region T is set as shown in FIG. 6B. More specifically, the 
region T is located at the lower portion of the image a. The 
vertical length of the region T is 30% of the vertical length 
of the image a, and the horizontal length of the region T is 
80% of the horizontal length of the image a. Note that the 
region T is located in the center of the horizontal length of 
the image a, with the lower end adjoining to the lower end 
of the image a. 

In the process of extracting corresponding points as 
shown in FIG. 5, matching process is repeated for a plurality 
of times with respect to the region T. A region (hereinafter 
referred to as a searching template) subjected to single 
matching processing is J /24 (='AxVb) of the region T. Asingle 
searching template is indicated by "t" in FIG. 6A. 

Steps S2 to S4 which will be described below are per- 
formed with respect to the region T extracted in step Si. 

In step S2, the corresponding-point-extracting unit 1 
determines a searching objective area S in the input image 
b, and within the area S, sets a searched area s which 
corresponds to the searching template t designated in the 
image a. The searching objective area S is set in the similar 
manner to the setting of the region T. That is, the searching 
objective area S as well as the searching template t are 
determined as shown in FIGS. 7A and 7B based on an 
assumption that, when a user takes photographs for gener- 
ating a panoramic image, the user would set the frames fl and 
£2 such that no more than 50% of the input image a overlaps 
the input image b in the horizontal direction, and no more 
than ±10% of the image a deviates in the vertical direction 
with the image b. More specifically, the template t which is 
a searched area s is set to have a size of 10 % in the vertical 
direction and less than 50 % (preferably, 40 %) in the 
horizontal direction, of the image a. 

Note that if an estimated overlapping condition of the 
input images a and b is different, the setting of the searching 
objective area for extracting corresponding points may be 
changed. 

The area S shown in FIG. 7B is the searching objective 
area S set in correspondence with the template t shown in 
FIG. 7A. The extracting unit 1 identifies a corresponding 
point in step S3. More specifically, the extracting unit 1 
parallelly moves the template t set in the image a within the 
searching objective area S set in the area b, i.e. the area s is 
moved within the area S. The extracting unit 1 then calcu- 
lates a summation of absolute values of difference between 
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all the image data included in the searching template t and 
all the image data included in the area s. The extracting unit 
1 determines a position in the image b having the minimum 
summation ZD of the absolute values of the differences, as 

s a corresponding point of the template t. 

In step S4, the corTesponding-point-extracting unit 1 
determines reliability of the result of the corresponding 
points detected in step S3. The determination of reliability is 
made on the basis of the minimum value ZD 1 ,^, and the 

10 second minimum value ID 2 min of the summation ZD of the 
absolute values of the differences. That is, assuming that Thj 
and Th 2 are the respective predetermined threshold values, 
the detected corresponding point is determined to be reliable 
when the following equations are satisfied: 

rD^STh, (1) 

£D 2 - i ,-£D 1 - I -Sra 2 (2) 

Then the corresponding-point-extracting unit 1 stores coor- 
20 dinates of the corresponding points for the images a and b 
respectively in a memory (not shown). 

Note that in order to detect positions of the corresponding 
points, besides from the above technique where the position 
having the minimum summation ZD of absolute values of 
25 the differences is determined as a corresponding point, the 
extracting unit 1 may calculate a correlation and determine 
a position having the largest correlation value as the corre- 
sponding point. 

In addition, the corresponding position may be deter- 
30 mined by a user. For instance, a user may designate an 
identical point in both images with a cursor or the like to be 
extracted, by referring to the two images a and b displayed 
on a display. 

In the foregoing manner, the overlapping region of the 
35 images a and b are determined by repeatedly performing the 
control steps described in FIG. 5. 

Determining Coordinates Conversion Parameter by Esti- 
mating Unit 2 

The parameter estimating unit 2 estimates parameters for 
d0 coordinates conversion on the basis of the extracted corre- 
sponding points from the overlapping region. Herein, coor- 
dinates conversion is the operation to coincide the overlap- 
ping region of the image a with the overlapping region of the 
image b. In the first embodiment, Affin transformation is 
45 employed. 

Assuming that the image b is rotated for 8° with respect 
to the position of the image a, parallelly moved for a distance 
dx in the direction of X and a distance dy in the direction of 
Y, and is enlarged m times as the image a, an arbitrary point 
50 ( x «» v a) in tnc image a corresponds to the point (x b , y^) in the 
image b defined by the following equation (3): 

i t -(cos e *„«ui By.-dx)xm-A x.+By.+C 
JV(-sin e x.+cos ey.-dy)xm—B x.*Ay.*D 




60 

C—mdx 

D—mdy (3) 

The parameter estimating unit 2 estimates the parameters 
65 A, B, C and D employing the least squares method. In order 
to estimate the parameters using the least squares method, at 
least two pairs of corresponding points are necessary. 
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Where only one pair of corresponding points is searched, 
the estimating unit 2 cannot perform matching processing 
since the rotation component 6 cannot be determined. In 
such case, the estimating unit 2 assumes 6=0 and m-1, 
which implies that the image is not rotated. In other words, : 
A and B arc set to 1 and 0, respectively. Thus, assuming that 
the corresponding points vectors is denoted as (a^, a^,), the 
unit 2 outputs the following parameters: 



8 



Aol(= cos 0), B=0(=sin 0) C— a„ D=- a y 



W : 



In a case where no corresponding points is obtained, the 
subsequent processing will not be performed; instead, for 
instance, a message or the like is outputted to a CRT and the 
processing ends. The parameters obtained in the above 
manner are used when an overlapping region is estimated. 15 

Tone Conversion 

In order to generate a combined image whose boundary of 
the images is inconspicuous, the tone converting units 4 and 
5 perform tone conversion on the inputted images a and b 
respectively so that the lightness and tonality of colors (R, 20 
G and B) are equal in the overlapping region of the two 
images. The correclion-coefficient-determining unit 3 deter- 
mines correction coefficients for the tone conversion pro- 
cessing. 

Meanwhile, the tone conversion on lightness components 25 
is made in the embodiment. Where the embodiment adopts 
color images, tone correction on each color (R, G and B) 
may be performed. Specifically, a tone conversion using 
individual tone correction function is made on each of R, G 
and B images. 30 

FIG. 8 shows the process algorithm performed by the 
correction-coefEcient-detennining unit 3 and the tone con- 
verting unite 4 and 5. The correction-coefficient-determining 
unit 3 inputs parameters A, B, C and D estimated by the 
estimating unit 2 for coordinates conversion, and the images 35 
a and b. The tone converting units 4 and 5 respectively input 
the images a and b. 

In step Sll of FIG. 8, the correclion-coefficient- 
determining unit 3 determines whether or not each pixel of 
the input images a and b is within the overlapping region. 4 
For determination, coordinates values of each pixel in the 
input images are subjected to Affin transformation according 
to the equation (3), utilizing the parameters A, B, C and D. 
Then determination is made as to whether or not the coor- 
dinates values on which Affin transformation has been 45 
performed are within the area determined to be the overlap- 
ping region of the image b. 

Next, in step S12 of FIG. 8, the correction-coefficient- 
determining unit 3 takes samples of image data for all or a 
predetermined number of pixels which are determined to be 50 
included in the overlapping region. Herein, it is assumed that 
N number of sample pixels are obtained. In other words, 
pixel values P„(k) and P 6 (k) (k=l to N) are obtained. The 
following equation (5) can be obtained if Affin transforma- 
tion in equation (3) is expressed simply by matrix H: 55 



(5) 



P*( x > y) represents a pixels value at (x, y) coordinate position 
which is not subjected to the Affin transformation. P 6 (x', y') 6 
represents a pixels value at (x, y) coordinate position which 
corresponds to the location (x 1 , y") that has been subjected to 
the Affin transformation. S of the equation (5) represents a 
function which performs lone conversion processing in 
pixels which is accompanied with the above coordinate 6 
transformation. The tone conversion processing will be 
described in detail later. 



Note that the oorrection-coefficient-determining unit 3 
may perform the sampling of pixel values pixel by pixel, or 
for every arbitrary number of pixels. Moreover, the 
correction-cc<ffiraent-dctcinjining unit 3 may utilize, as 
sample data, an average value of the neighboring pixel 
values based on the coordinates of the corresponding points 
obtained by utilizing the parameters. 

The correction-coefficient-determining unit 3 then obtains 
tone correction coefficients in step S13 of FIG. 8 on the basis 
of the sample data P„(k) and P t (k). 

FIG. 9 shows the brief method of calculating tone cor- 
rection coefficients. 

Referring to FIG. 9, the abscissa indicates a density value 
of the sample pixel data P 6 (k) of the image b; and the 
ordinate, a density value of the sample pixel data P„(k) of the 
image a. In step S13 of FIG. 8, the correction-coefficient- 
determining unit 3 generates a conversion function (function 
fl>00 in FIG. 9) on the basis of the above sample data, to 
coincide the pixel density value of one image (e.g. image a) 
to the pixel density value of the other image (e.g. image b). 

Experimentally speaking, density values of the images a 
and b have a distribution similar to a quadratic function. 
Thus, to convert a pixel value in the overlapping region of 
the image b to a pixel value in the overlapping region of the 
image a, the following quadratic function will be employed. 

fiPA-T bl xP?+T b #P h +T u (6) 
where T M , T b2 and T b3 are coefficients. 

In step S13 of FIG. 8, the correction-coefficient- 
. determining unit 3 obtains the coefficients T M , T w and T i3 
to generate the f(P fc ) in equation (6). To obtain these values, 
the coirection-coefEcient-determining unit 3 calculates T fcl , 
Tj^ and T b3 which minimizes an evaluation function e 
expressed by the following equation: 

^{pJkyiT^^T^p^T^)) (7) 

The correcuon-coefficient-determining unit 3 supplies the 
tone converting units 4 and 5 with the calculated coefficients 
Tm« T*2 and T^. Note that since the correction-coefficient- 
determining unit 3 calculates the coefficients to coincide the 
pixel values of the image b with the pixel values of the image 
a in the first embodiment, the tone correction coefficients 
Ton T.2 an d T,3 for the image a are respectively, T al =0, 
T a2 =l and T a3 =0. 

Next, the tone converting units 4 and 5 convert pixel 
values of each of the images a and b in accordance with the 
tone correction coefficients T w , T fc2 and T M in step S14 of 
FIG. 8. Hereinafter, the operation performed by the tone 
converting unit 5 will be described. 

The tone converting unit 5 converts tones of the image b 
into tones of the image a. The tone converting unit 5 
generates a table for converting the tones of the image b to 
the tones of the image a on the basis of the tone correction 
coefficients T M , T M and T M . When the dynamic range of an 
image is 8 bits, pixel values 0 to 255 of the image b have the 
values f(0) to f(255) in the space of the image a, according 
to the quadratic function f(P 6 ) in equation (6). Examples of 
f(0) to f(255) in a conversion table 910 are shown in FIG. 10. 

Although the tone converting unit 4 is capable of con- 
verting density values of the image a into density values in 
the space of image b, pixel values of the image a do not need 
to be converted in the first embodiment; thus f(PJ=P a . 
Therefore, the conversion table in the tone converting unit 4 
converts the pixel values 0 to 255 into pixel values 0 to 255, 
in other words, no conversion is performed. 

Note that in a case of a color image, it is preferable to 
perform tone conversion by generating a conversion func- 
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lion commonly utilized by R, G and B. Tone 
functions for respective colors may be provided so that color 
matching may be improved. 

Although the quadratic function is utilized as a conversion 
function in the first embodiment, it is possible to utilize s 
another form of function. Needless to say, it is also possible 
to perform tone conversion by utilizing a non-linear table. 

Combining Images 

Image combining unit 6 generates a single combined 
image, on the basis of the input images and the images on 10 
which the tone correction has been respectively performed 
by the tone converting unit 4 and 5. 

The image combining unit 6 generates a combined image 
c in accordance with an algorithm shown in FIG. 11. 

In step S21, the image combining unit 6 sets an image area 15 
for the combined image c. 

The area indicated by broken lines in FIG. 12 denotes the 
area for the combined image which has been set on the basis 
of a coordinates system of the input image a. Since the 
description of the first embodiment bases upon the coordi- 20 
nates of the input image a, for the purpose of a simple 
explanation, the upper side and the lower side of the image 
a are assumed to be parallel to coordinates axis X. 

The image combining unit 6 first coincides the left end of 
the combined image c with the left end of the image a. The 25 
image combining unit 6 converts a position of the pixel 
(100) at the upper right end of the image b and a position of 
the pixel (101) at the lower right end of the image b 
respectively to positions in the coordinates system of the 
image a. Then, the image combining unit 6 coincides the 30 
right end of the combined image c with a position having a 
larger coordinates value between the converted two posi- 
tions. In FIG. 12, since the lower right end 101 has a larger 
coordinates value, the image combining unit 6 coincides the 
right end of the combined image c with the lower right end 35 
101. 

Note that for the process of converting a coordinates value 
of image b into a coordinates value of image a to obtain a 
position having a larger coordinates, the image combining 
unit employs the inverse transformation of Affin transfer- 40 
mation shown in FIG. (8). 



C^-A-C+B-D^+B 2 ) 50 
D^i-B-C-ADy^+B 2 ) (g) 

Herein, A', B', C and D' are parameters of the inverse 
transformation. 55 

Similarly (see FIG. 12), the upper end of the combined 
image c is determined in the following manner. More 
specifically, the position of the pixel (100) at the upper right 
end of the image b and the position of the pixel (103) at the 
upper left end are converted respectively to positions in the 60 
coordinates system of the image a. The image combining 
unit 6 coincides the upper end of the combined image c with 
a position having the smallest coordinates value among the 
two converted positions, and the position of the pixel (102) 
at the upper left end of the image a. Furthermore, the image 65 
combining unit 6 coincides the lower end of the combined 
image c with a position having a larger coordinates value 
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between the position of the pixel (101) at the lower right end 
and the position of the pixel (105) at the lower left end of the 
image b. 

The combined image area c shown in FIG. 12 is deter- 
mined in the above-described manner. 

In step S22 in FIG. 11, the image combining unit 6 sets a 
boundary of the images so that the boundary is the center of 
the overlapping region. Since the images a and b are 
arranged side by side in the first embodiment, the boundary 
L which has been set in the center is indicated by broken 
lines L in FIG. 12. To explain more in detail, the boundary 
L is set parallel to vertical coordinates axis Y. In the 
horizontal direction, the boundary L is set so that the two 
images are combined at a barycenter between the coordi- 
nates of the pixel 106 located at the lower right end of the 
image a and a smaller coordinates value of the two pixels 
103 or 105, respectively located at the upper left end and 
lower left end of the image b, which have been converted 
respectively to pixel positions in the coordinates system of 

In step S23, the image combining unit 6 repeatedly 
performs the processing in steps S24 and S25 on the area for 
the combined image c set in step S21. 

The present invention has an object to generate a com- 
bined image whose density difference at the boundary of the 
images is inconspicuous. This object is partially attained by 
the tone correction in step S14 in FIG. 8. However, the tone 
conversion performed in step S14 is performed indepen- 
dently for each of the input images a and b, therefore does 
not always minimize the density difference near the bound- 
ary of the images in the overlapping region. The processing 
performed in steps S24 and S25 further smoothes the density 
difference near the boundary of the images in the overlap- 
ping region. 

As shown in FIG. 13, a tone conversion area 150 having 
a predetermined width 2W is set in steps S24 and S25 < 
FIG. 11, with the boundary L of the images as its center. 1 
other words, the tone conversion area 150 is a rectangul: 
area having a width W respectively to the left and right ol 
the boundary L. 

In step S24, the image combining unit 6 writes pixel 
values of the pixels of the image a in the corresponding area 
of the combined image area c. With respect to those pixels 
included in an area 140a of the image a shown in FIG. 13 
but not included in the tone conversion area 150, the image 
combining unit 6 writes the pixel values of the original 
image a. With respect to a pixel ¥ a in the image a included 
in the tone conversion area 150, the image combining unit 
6 determines a tone conversion value P'„ in accordance with 
how far (dx„) the pixel P„ is from the boundary L. The tone 
conversion value P' 0 is determined by the following 
equation, utilizing a function f defined in the above- 
described conversion table (FIG. 10) stored in the tone 
converting unit 4. 



(9) 



According to equation (9), the tone 
s obtained by adding a correction term 
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plurality of areas may be set in the overlapping region of 
p _ <i*= each image (150L and 150R), and weights may be added to 
" w w the pixel values of each image in each of the plurality of 
areas to generate pixel values of a combined image. 
,. . ... , , , , v s The combmed image c generated by me foregoing pro- 
corresponding to a distance dx„ is added to the value f(P„) cessing is outputted to a printer or the like 
which has been corrected in accordance with a function f in Modification of First Embodiment, 
the conversion table. ... First Modified Embodiment 

As mentioned above, since the tone converting unit coin- A modification of the first embodiment (first modified 

cides the tone of the image b with that of the image a in the 1Q embodiment) will be described next, 
first embodiment, f-1. Accordingly, equation (9) is An image combining apparatus according to the first 

expressed by the following equation: modified embodiment has the same arrangement as that of 

p . _ p the first embodiment shown in FIG. 4. However the func- 
tions and operation of the correction-coeflBcient-determining 

In step S25, the image combining unit 6 similarly writes unit 3 and the image combining unit 6 are different from 

pixel values with respect to the area of image b. That is, with 15 those of the first embodiment. The operation will be 

respect to those pixels included in an area 140fc of the image described hereinafter. 

b shown in FIG. 13 but not included in the tone conversion The correction-coefEcient-determining unit 3 according 

area 150, the image combining unit 6 writes the pixel value t 0 the first modified embodiment determines pixel values 

of .the original image b. With respect to a pixel P b in the included in the tone conversion area 150 on the basis of pixel 

image b included in the tone conversion area 150, the image 20 values in the overlapping region of the input images a and 

combining unit 6 determines a tone conversion value P' fc in b. The technique of determining thereof will be described 

accordance with how far (dx fc ) the pixel P b is from the with reference to FIG. 16. 



boundary L. The tone conversion value P b is determined by As similar to the first embodiment, the correction- 
the following equation, utilizing a function f defined in the coefficient-determining unit 3 generates sample data P (k) 
conversion tahlr. <dnn-H in the irw ranwriiu iimi < 2S j n m ... . . .... „ . . ' 



n table stored in the tone converting unit 5. 25 and P fc (k) of corresponding points on the basis of the pixel 
values (see FIG. 16) included in the overlapping region of 
P b = Pb .'ti. + /(p^fi.o-^) toe image a and b. When generating the sample data, the 

w V w ) correction-coefficient-determining unit 3 calculates an aver- 

3Q age value dP avg of difference dP in each pair of correspond- 
FIG. 14 shows the combined image c obtained in the mg P ° intS * and -"W^ the average value dP to the image 
above-described manner. combining unit 6 Note that m FIG. 16, the reference letter 

A dummy area 160 indicated by hatching in FIG. 14 is a * dea ° teS 3 f ™ dth ° f thc mage a ' ^ ima 8 e b has ihe 

region where pixels of neither the image a nor image b are Sam6 * C " ^ . c 

written. In the dummy area 160, a dummy pixel (e.g. white 35 (l , Assumin S that the dynamic range of apixel value is 8 bits, 
pixel or the like) is written lmage combmm g 111111 6 sets the tone conversion area 2W 

FIG. 15 shows the variation of pixel values at an arbitrary ° n baSiS ° f P «* by ^ Mlo ™S equation: 
line H in the combined image c shown in FIG. 14. In the 

graph in FIG. 15, the drop 170 of the pixel value P„ iw=Yx^- (l2) 

represents a dark subject; and the rise 171 of the pixel value 40 235 
Pi, a light subject. 

The variance in the graph in FIG. 15 will be described dP avg in equation (12) denotes the difference in tones 

next. Since the pixel values of the image a are written in the between each pair of corresponding points in the overlap- 
combined image area c without being processed in the first ping region of the images, having a maximum value 255 and 
embodiment, the pixel values located in the left side of the 45 a minimum value 0. According to equation (12), when the 
boundary L are the pixel values of the original image a. In difference of the tone levels is large, the width 2W in the 
other words, as expressed by equation (10), the pixel values tone conversion area becomes large; and when the difference 
located in the left side of the boundary L are not converted of the tone levels is small, the width 2W becomes small, 
even though the pixels are within the tone conversion area The foregoing description explains the technique of deter- 

150 - . 50 mining the tone conversion area according to the first 

Meanwhile, with respect to the pixel values located in the modified embodiment, more specifically, the method of 
right side of the boundary L, tone conversion performed on determining a width of the tone conversion area in corre- 
each pixel would be different depending upon whether the spondence with difference in tone levels, in a case where 
pixel is within the tone conversion area 150R (width W). tones of the image a are different from tones of the image b 
More specifically, with respect to a pixel value 133 included 55 in the overlapping region. 

in the tone conversion area 150R, the pixel value is con- According to thc first modified embodiment, a combined 

verted by the equation (11) such that the value gradually image is generated in the same manner as that of the first 
changes from the pixel value 131 of the original image b to embodiment 

the pixel value 132 converted in accordance with the cor- According to the first modified embodiment, since the 

rection table (function f ) stored in the tone converting unit 60 tone conversion area can be suitably set in correspondence 
^- with the difference in tone levels in the overlapping region 

With respect to the area outside the tone conversion area of the images, it is possible to perform smooth tone correc- 
150R, pixel values of the original image b are written. lion which is appropriate for an inputted image. 

In the foregoing description, the image combining unit 6 Modification of First Embodiment 
performs writing of pixel values for the area of image a 65 ... Second Modified Embodiment 
separately from the area of the image b, dividing the The arrangement of an image combining apparatus 

combined image area by the boundary L. However, a according to thc second modified embodiment is identical to 
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that of the first embodiment (FIG. 4). Since the method of 
tone correction alone is different from that of the first 
embodiment, the method will be described below. 

The image combining apparatus according to the second 
modified embodiment extracts sample data on the basis of s 
pixel values in the overlapping region of input images a and 
b. This operation is identical to that of the first embodiment. 
The correlation between a pair of pixel values P a and P b 
which are extracted as sample data is determined in accor- 
dance with the method shown in FIG. 9. 

In the first embodiment, the conversion function f900 10 
(FIG. 9) is generated to coincide tones of the image b with 
tones of the image a on the basis of the relationship of each 
sample data by utilizing the least squares method. 
Accordingly, in the first embodiment, conversion of pixel 
values performed for tone correction is executed only on the 15 
image b. In the second modified embodiment, tone correc- 
tion is executed on pixel values P„ and P 6 of each image. 

To describe more in detail, the correction-coefficient- 
determining unit 3 according to the second modified 
embodiment generates a conversion function f as similar to 20 
the technique described in the first embodiment with refer- 
ence to FIG. 9, and provides the tone converting units 4 and 
5 with tone correction coefficients (T M , T b2 and T fc3 ) which 
prescribe the function f. Note that the conversion function f 
serves to coincide pixel values of image b with pixel values 2 5 
of image a, as similar to the first embodiment. 

The tone converting unit 5 for converting the input image 
b performs tone conversion utilizing the conversion function 
f on the basis of the pixel values of image b. The tone 
conversion is executed by the following equation (13), 30 
where P t denotes a pixel value of the original image b, and 
f(P t ) denotes a pixel value converted by the conversion 
function. 




(13) 35 



In other words, the image combining apparatus according 
to the second modified embodiment performs tone conver- 
sion on the input image b, by utilizing an average value of 
the original pixel value P fc and the pixel value f^PJ obtained 
by the conversion function f. 

Meanwhile, the tone converting unit 4 performs tone 
conversion in accordance with the following equation (14): 

P -p . ^ p - ) - p " < M > 



where P„ is a pixel value of the original image a. 50 

FIG. 17 shows the brief process of tone conversion 
performed on the image b in accordance with equation (14). 
As shown in FIG. 17, {f(P fc )-P 6 } denotes an offset amount 
between the conversion function f(P fc ) and the original pixel 
value P 6 . Equation (14) expresses that tone conversion is 55 
performed by adding/subtracting half the value of {f(Pt)- 
P 6 } to/from the images a and b. By virtue of this calculation, 
pixel values of each image gradually change, thus smooth 
tone correction is realized. 

The pixel values converted by each of the tone converting 60 
units 4 and 5 are maintained in a form of a table as shown 
in FIG. 10, as similar to the first embodiment. 

In the second modified embodiment, the image combining 
unit 6 performs writing of pixel values in the combined 
image c, as similar to that described in the first embodiment. 65 

FIG. 18 shows variation of pixel values on the line H in 
FIG. 14 with respect to the image combined by the second 
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modified embodiment. It is apparent from the graph that 
pixel values of each image are gradually changed, therefore 
smooth tone correction is realized. 
<Second Embodimcnt> 

FIG. 19 is a block diagram showing the arrangement of an 
image combining apparatus 210 and the process flow of an 
inputted image according to the second embodiment. 

Referring to FIG. 19, the image combining apparatus 210 
mainly includes: input terminals 210a and 2106; a 
corresponding-point-extracting unit 220 which extracts cor- 
responding points from two input images; a corresponding- 
point-selecting unit 225 which removes components which 
are erroneously detected as corresponding points from a set 
of corresponding points extracted by the corresponding- 
point-extracting unit 220, and which selects only the correct 
corresponding points; a parameter estimating unit 240 which 
estimates parameters for combining images; a correction- 
coefBcient-determining unit 230 which determines coeffi- 
cients for tone correction of the image; tone converting units 
231 and 232 which respectively correct tones of input 
images a and b; and an image combining unit 250 which 
combines two images on which tone conversion has been 
performed. 

Images a and b are respectively input to the input termi- 
nals 210a and 2106. The input images a and b are picked tip 
by an electronic still camera or a video camera or the like in 
a manner such that both images partially include an over- 
lapping region having the same image. As similar to the first 
embodiment, the exposure conditions at the time of image 
pick-up are different for the two images. 

Each of the input terminals 210a and 2106 is connected to 
the corresponding-point-extracting unit 220, correction- 
coefBcient-determining unit 230, and tone converting units 
231 and 232. The input images a and b are seDt to the 
corresponding-point-extracting unit 220 where correspond- 
ing points are extracted from each of the images a and b. 

The corresponding-point-extracting unit 220 is connected 
to the corresponding-point-selecting unit 225 and the param- 
eter estimating unit 240. The corresponding points extracted 
by the corresponding-point-extracting unit 220 are sent to 
the corresponding-point-selecting unit 225 and the param- 
eter estimating unit 240. The corresponding-point-selecting 
unit 225 and the parameter estimating unit 240 are interac- 
tively connected to each other. On the basis of coordinates 
conversion parameters estimated by the parameter estimat- 
ing unit 240 and a coordinates value indicative of the 
corresponding points extracted by the corresponding-point- 
extracting unit 220, the corresponding-point-selecting unit 
225 removes components which are erroneously detected as 
corresponding points from the corresponding points 
extracted by the corresponding-point-extracting unit 220, 
and sends the result to the parameter estimating unit 240. 
The parameter estimating unit 240 estimates parameters for 
image conversion on the basis of the result where erroneous 
corresponding points have been removed. 

The parameter estimating unit 240 is connected to the 
correction-coefficient-determining unit 230 and image com- 
bining unit 250. The image conversion parameters estimated 
by the parameter estimating unit 240 is sent to the 
corrcction-coeffitient-determining unit 230 and the image 
combining unit 250. 

The correction-coefficient-detennining unit 230, which 
holds images a and b inputted by the input terminals 210a 
and 2106 and image conversion parameters (A, B, C and D) 
estimated by the parameter estimating unit 240, determines 
coefficients (T„ and T b ) for tone correction of an image on 
the basis of the image included in the overlapping region of 
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the images a and b. The correction-coefficieni-delermining virtue of this process, corresponding points, which will be 

unit 230 is connected to the tone converting units 231 and utilized by the parameter estimating unit 240 for estimating 

232. The correction-coefficient-determining unit 230 sends image conversion parameters, are narrowed down to more 

the determined coefficients (T 0 and T fc ) to the tone convert- appropriate values. The template matching, performed by 

ing units 231 and 232, respectively. The tone converting 5 the extracting unit 220 to extract corresponding points, has 

units 231 and 232 perform tone conversion on each of the a possibility of erroneous recognition. Therefore, the corre- 

images a and b utilizing the coefficient (T 6 ) sent by the sponding points must be reconfirmed by a method different 

correction-coefficient -determining unit 230, so that lightness from template matching. 

is equal in the overlapping region of the images a and b. FIG. 20 illustrates processing performed by the selecting 

Each of the tone converting units 231 and 232 is con- 10 unit 225 where determination is made as to whether or not 

nected to the image combining unit 250. The image com- the corresponding points are erroneously recognized. In 

billing unit 250 converts the images a and b, on which tone FIG. 20, it is assumed that a set of plural pairs of points P,(x, 

correction has been performed respectively by the tone y) and Q/x', y 1 ) (i-0, 1, ...) is a set of corresponding points 

converting units 231 and 232, by utilizing the parameters of images a and b. The corresponding-point-selecting unit 

estimated by the parameter estimating unit 240 and com- 15 225 utilizes the Affin transformation parameters A, B, Cand 

bines the two images into a single image. D obtained by the parameter estimating unit 240 to detect 

In the foregoing arrangement, the control unit 270 con- components of erroneous corresponding points in accor- 

trols the entire image combining apparatus. dance with the following equation: 

As will be described later, the second embodiment largely W-(a x *B- +OI<37i 

differs from the first embodiment by including the 20 x y < 

corresponding-point-selecting unit 225. \y>-(.-B-x<A-y+D)\<Th (15) 

Hereinafter, the operation of the image combining appa- Inequation (15), (A-x+B-y+C) and (-B-x+A-y+D) respec- 

ratus 210 will be described in a case where a combined tively indicate coordinates (x, y) of an arbitrary point 

image c is generated based on the images a and b. Note that (estimated to be one of corresponding points) on the image 

in the second embodiment, the input images a and b each 25 a on which Affin transformation has been performed, 

includes density image data having N number of tones. Accordingly, if an absolute value of the difference between 

Extracting Corresponding Points the coordinates (A-x+B-y+C, -B-x+A-y+D), on which Affin 

The input two images a and b are first sent to the transformation has been performed on (x, y), and coordi- 

corresponding-point-extracting unit 220 where correspond- nates (x 1 , y') on the image b, is larger than a threshold value 

ing points are extracted. 30 Th, (i.e. if equation (15) is not satisfied), the pair of 

The processing for extracting corresponding points per- corresponding points P,{x, y) and Q^x', y') is considered to 

formed by the corresponding-point-extracting unit 220 is be erroneously recognized. 

substantially identical to the processing performed by the The corresponding-point-selecting unit 225 selects only 

extracting unit 1 in the first embodiment. In other words, the the corresponding points that satisfy equation (15) from the 

extracting unit 220 in the second embodiment executes 35 set of corresponding points obtained by the corresponding- 

substantially the same processing steps as the extraction point-extracting unit 220. 

processing steps (FIG. 5) in the first embodiment. That is, Referring to FIG. 20, arrows indicated by solid lines and 

the extracting unit 220 extracts a set of corresponding points broken lines represent movement of positions of the points 

by performing template matching. according to Affin transformation. For instance, a pair of 

Estimating Coordinates Conversion Parameters ■ «o corresponding points indicated by the solid fine, e.g. a pair 

The corresponding -point-extracting unit 220 sends the of points P 0 and Q 0 or a pair of points P 2 and Q 2 is selected 

extracted corresponding points to the parameter estimating as the pair of corresponding points because the position 

unit 240 and the corresponding-point-selecting unit 225. obtained after Affin transformation and a position assumed 

The parameter estimating unit 240 estimates parameters to be the corresponding point are substantially equal. On the 

(A, B, C and D) for coordinates conversion, on the basis of 45 other hand, a pair of corresponding points P 3 and Qj or a pair 

the extracted corresponding points. of points P 3 and Q 3 obtained by the corresponding-point- 

Thc second embodiment also requires coordinates con- extracting unit 220 is deviated for more than a threshold 

version to coincide the overlapping region of the image a value, from the coordinates obtained on the basis of afore- 

with the overlapping region of the image b. Thus, Affin mentioned parameters. These corresponding points are 

transformation is employed as similar to the first embodi- 50 removed as erroneously recognized components, 

ment. Accordingly, the parameter estimating unit 240 Referring back to FIG. 19, the corresponding-point- 

according to the second embodiment determines parameters selecting unit 225 sends the result of selection to the 

A, B, C and D to be applied to the Affin transformation parameter estimating unit 240. The parameter estimating 

expressed by equation (3), and outputs the result to the unit 240 estimates parameters used for combining images by 

correction-coefficient-determining unit 230. In other words, 55 utilizing the selected corresponding points. As set forth 

the estimating unit 240 according to the second embodiment above, parameters for combining images are estimated on 

is substantially identical to the estimating unit 3 according to the basis of coordinates of corresponding points from which 

the first embodiment. erroneous components have been removed. Therefore, the 

Selecting Corresponding Points image combining apparatus can generate a more precise 

A set of corresponding points extracted by the 60 combined image. 

corresrx>nding-point-extracting unit 220 and coordinates Tone Correction Processing 

conversion parameters (A, B, C and D) estimated by the The parameter estimating unit 240 sends the estimated 
parameter estimating unit 240 are input to the parameters (A, B, C and D) to the correction-coefficient- 
corresponding-point-selecting unit 225. The corresponding- determining unit 230. The correction-coefficient- 
point-selecting unit 225 removes erroneously detected cor- 65 determining unit 230 determines correction coefficients to be 
responding points from the set of corresponding points utilized by the tone converting units 231 and 232 for tone 
extracted by the corresponding-point-extracting unit 220. By correction. 
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FIG. 21 is a flowchart showing the tone correction pro- pixel f a (P ak ). The coefficients (T M , T w and T M ) of the tone 

cessing performed by the correctioQ-coefficient-determining correction function f fc are calculated by the least squares 

unit 230 and tone converting units 231 and 232. method, utilizing the sample pixels shown in FIG. 23B, to 

Sample pixels are first obtained from images a and b to minimize e in equation (17). 

determine tone correction coefficients (step S31). s 

FIG. 22 illustrates obtained sample pixels. The correction- ^.(f^HJ^xp^r^^T^Y (17) 

coefficient-determining unit 230 first determines on the basis ,„ „ ,„ „ e 

of the image combine parameters (A, B, C and D) estimated JllZ^ ™* ^ ^gomg processing, tone cor- 

by the parameter estimating unit 240, as to whether or not a ™* 4 for converting tones of the .mages 

coordinates value of a sample pixel P,,, of the image a is ,o * ^ " "* by ^ f ° ll0Wmg eqUaU ° n ^ 

within the overlapping region. To make determination, the /.('V., -P+T^i+T., 

correction-coefficient-determining unit 230 converts the . 2 

coordinates value of a sample pixel P al of the image a into fM-Tu-i +T a i+T b3 ( 1S ) 
a coordinates value of image b by Affin transformation where i denotes tones before conversion. The correction- 
utilizing equation (3), and makes determination as to is coefficient-determining unit 230 sends the calculated tone 
whether or not the converted coordinates value is included correction coefficients T„ and T 6 respectively to the tone 
in the image b. If it is determined that the sample pixel P al converting units 231 and" 232. 

is in the overlapping region of the images a and b, a Referring back to the flowcharts in FIG. 21, the tone 

coordinates value corresponding to the sample pixel P„, in converting units 231 and 232 respectively convert each of 

the image b is set to be a sample pixel P M . The correction- 20 the pixel values of images a and b utilizing each of the tone 

coefficient-determining unit 230 performs the same process correction coefficients sent by the correction-coefficient- 

on a plurality of sample pixels P o1 to P^ (k =1 to N) of the determining unit 230 (step S34). 

input image a, and the obtained pixel values will be set as The operation of the tone converting unit 232 is described 

sample pixels P M to P bk (k=l to N). below. Note that the tone converting unit 231 executes the 

Note that, as similar to the first embodiment, the 25 same operation as that of the tone converting unit 232. 
correction-coefficient-determining unit 230 may perform the In the tone converting unit 232, a table is generated to 
sampling of pixel values pixel by pixel, or for every arbitrary convert tones of the image b on the basis of tone correction 
number of pixels. Moreover, the correction-coefficient- coefficients T fcl , T^ and T b3 . More specifically, assuming 
determining unit 230 may utilize, as sample data, an average that the dynamic range of the image is 8 bits, a table as 
value of neighboring pixel values based on the coordinates 30 shown in FIG. 24 is generated to convert a pixel value (0 to 
of the corresponding points obtained by utilizing the image 255) of the image b into a pixel value £ b (0)~f b (25S) accord- 
combine parameters. ing to the quadratic function in equation (18). 

Referring back to FIG. 21, the correction-coefficient- Note that in a case of a color image, the tone, converting 
determining unit 230 obtains tone correction coefficients of unit may perform tone conversion for each of the colors R, 
the image a on the basis of sample pixels P ak and V bk 35 G and B. Alternatively, the tone converting unit may perform 
obtained in the foregoing manner (step S32). The tone tone conversion by generating a conversion function corn- 
correction coefficients are calculated as a coefficient of a monly used for the colors R, G and B in accordance with a 
tone correction function f„. luminance signal. In this regard, the functioD f„ which 
The method of performing tone correction according to corrects tonality of the image a may be generated on the 
the second embodiment will be described with reference to 40 basis of pixel values of G image signals. All the R, G and B 
FIG. 23A. In FIG. 23A, the abscissa indicates tones of an components of the image a may be converted by the function 
image before correction, and the ordinate indicates tones f 0 . Conversion functions f b for R, G, B colors of the image 
after correction. b are generated in such a manner that pixel values of each 
First, average values AvgfPJ and Avg(P fc ) of the sample color component of the converted image b by f 6 may match 
pixels P^ and P^ of the images a and b are calculated, and 45 with those of the image a converted by using the function f„. 
difference D of each average value is calculated by the Furthermore, although the quadratic function is utilized in 
following equation: the above description, another form of function may be 
£>=Avg(P>Avg(/> t ) ns) utilized. It is also apparent that tone conversion may be 
performed by utilizing a non-linear table. 
Then, as shown in FIG. 23A, a correction function f„ is 50 Combining Images 
determined so that the tones of the image a can be expressed The tone converting units 231 and 232 sends the images 
by a quadratic function (tone correction function) passing a and b, on which tone conversion has been performed, to 
through three points: (0, 0), (255, 255) and (128, 128+D/2). the image combining unit 250 where the images are com- 
Herein, since an arbitrary quadratic function can be defined bined into a single image. Since the process of combining 
by three parameters, it is assumed that the tone correction 55 images is identical to that of the first embodiment (FIG. 11), 
function f„ is defined to be T o1 , T a2 and T o3 . In addition, a detailed description thereof will be omitted, 
value obtained by converting the sample pixel P ak of the As has been described above, according to the image 
image a utilizing the tone correction function f„, is defined combining apparatus of the second embodiment, errone- 
to be f^P^. ously detected corresponding points are first determined in 
Referring back to the flowchart in FIG. 21, upon calcu- 60 an overlapping region of images a and b partially having the 
lating the coefficients (T al , T„ 2 and T a3 ) of the tone correc- - same subject. Then, on the basis of a result of extracted 
tion function f„ of the image a, the correction-coefficient- corresponding points from which the erroneously detected 
determining unit 230 calculates coefficients (T M T w and corresponding points are removed, parameters (A, B, C and 
I™ ° f 1000 correctio11 f"" 1 ^ 011 f * of the ™age b (step D) for combining the images are determined. Next, sample 

65 pixels (P ak and P^) used for tone correction are generated, 

The tone correction function f 6 is obtained to be utilized utilizing the image combine parameters, and a tone correc- 

as a function to coincide the sample pixel P M with sample tion coefficient (TJ of the image a is calculated on the basis 
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of an average value of the sample pixels. On the basis of the 
sample pixels of the image a on which tone correction has 
been performed utilizing the tone correction coefficient, a 
tone correction coefficient (T b ) of the image b is calculated. 
Since the images a and b are combined after tone correction s 
is performed on the images a and b utilizing each of the tone 
correction coefficients calculated in the foregoing manner, it 
is possible to generate a combined image whose boundary of 
the images is inconspicuous, even in a case where the two 
images have different exposure conditions. 

Furthermore, on account of the tone conversion of the 10 
image a performed on the basis of an average level of the 
pixel values in the overlapping region of the images, which 
is obtained in accordance with the above described image 
parameters, and also on account of the correction performed 
to coincide the tones of the image b with the corrected tones 15 
of the image a, the corrected tones of the images a and b are 
natural. 

<Modification of Second Embodiment- 
. . . Third Modified Embodiment 

Next, a modification of the second embodiment (third 20 
modified embodiment) will be described with reference to 
FIGS. 25Ato26B. 

An image combining apparatus according to the third 
modified embodiment has the same arrangement as that of 
the second embodiment shown in FIG. 19. However, func- 25 
tions and operation of the corresponding-point-extracting 
unit 220, corresponding-point-selecting unit 225 and 
correction-coefficient-determining unit 230 are different 
from those of the second embodiment. Hereinafter, descrip- 
tion thereof will be provided. 

Unlike the corresponding -point-extracting unit of the sec- 30 
ond embodiment, the corresponding-point-extracting unit 
220 of the image combining apparatus 210 according to the 
third modified embodiment does not determine (step S4 in 
FIG. 5) reliability of corresponding points obtained by 
matching operation. Instead, the third modified embodiment 35 
recognizes a position having a minimum value of non- 
coincidence as a corresponding point. 

The corresponding-point -selecting unit 225 repeats the 
process of removing erroneous corresponding points which 
is shown in the aforementioned equation (15), thereby 40 
enhancing accuracy in extracting corresponding points. 

FIGS. 25A-25C are explanatory views for explaining the 
steps of removing erroneous corresponding points per- 
formed by the corresponding-point-selecting unit 225. 

In accordance with a result of extraction (FIG. 25A) 45 
obtained by the corresponding-point-extracting unit 220, the 
parameter estimating unit calculates image conversion 
parameters A, B, C and D. The difference of each pair of 
corresponding points is then obtained in accordance with 
equation (15). Next, average values Avg(dx) and Avg(dy) of so 
the differences are calculated. In the third modified 
embodiment, the average value of the differences is used as 
a threshold value. More specifically, the corresponding- 
point-selecting unit 225 removes erroneously recognized 
corresponding points as shown in FIG. 25B utilizing the 55 
values Th^-Avg(dx) and Th J ,-Avg(dy). 

The corresponding-point-selecting unit 225 repeats the 
above processing until the average values Avg(dx) and 
Avg(dy) of the differences are converged to a predetermined 
value. The resultant corresponding points (FIG. 25C) are 60 
sent to the parameter estimating unit 240 where image 
combine parameters are estimated. 

As similar to the second embodiment, the correction- 
coefficient-determining unit 230 obtains tone correction 
coefficients T„ and T b on the basis of the sample pixels 65 
and P^. obtained in the steps S32 and S33 in the flowchart 
in FIG. 21. 
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FIGS. 26A and 26B are graphs for explaining the steps of 
determining tone correction coefficients by the correction- 
coefErient-determining unit 230. 

More specifically, the correction-coefficient-determining 
unit 230 calculates the average values Avg(P„) and Avg(P 6 ) 
of the sample pixels P^ and P tt obtained from each of the 
images a and b, and further calculates difference D of each 
of the average values. As shown in FIG. 26A, tones of the 
image a are converted such that the tones are expressed by 
the linear function f„' passing through the points (0, 0) and 
(128, 128+D/2). The sample pixel V ak of the image a is 
converted in accordance with the tone correction function 
f„', and the converted sample pixel is defined to be f„' (P^. 
Note that the coefficients of the tone correction function f„' 
are l ai ' and T o2 '. 

Furthermore, tone correction coefficient f fc ' is generated to 
coincide the sample pixel P^. with the sample pixel f„' (P ai ) 
The abscissa in FIG. 26B indicates tones before correction, 
and the ordinate indicates tones corrected by utilizing the 
tone correction coefficient f 6 '. 

The coefficients T M , T fc2 and T b3 utilized to generate the 
tone correction function f 6 ' are calculated by the least 
squares method utilizing the sample pixels shown in FIG. 
26B, to minimize € in equation (17). 

In accordance with the foregoing processing, tone cor- 
rection function f„' and f fc ' for converting tones of the images 
a and b are obtained by the following equation (19): 



(19) 

where i denotes tones before conversion. 

As set forth above, according to the third modified 
embodiment, the corresponding-point-extracting unit does 
not determine the reliability of the corresponding points 
obtained by matching processing. Instead, a position having 
a minimum value of non-coincidence is recognized as a 
corresponding point. By virtue of this processing, it is 
possible to improve accuracy in extracting corresponding 
points. In addition, image combine parameters are set in 
accordance with the corresponding points having high pre- 
cision from which erroneous corresponding points are 
removed. Further, the sample pixels for tone correction are 
generated by utilizing the image combine parameters. 
Therefore, it is possible to generate a combined image 
whose boundary of the images is inconspicuous, even in a 
case where the two images have different exposure condi- 

<Modification of Second Embodiment 

... Fourth Modified Embodiment 

The fourth modified embodiment will be described next 
with reference to FIG. 27. 

The arrangement of the image combining apparatus 
according to the fourth modified embodiment is identical to 
that of the second embodiment shown in FIG. 19. However, 
the operation of the correction-coefficient-determining unit 
230 is different from that of the second embodiment. Here- 
inafter the operation thereof will be described. 

In the foregoing second embodiment and the third modi- 
fied embodiment, the correction-coefficient-determining 
unit obtains the sample pixels V ak and P^ shown in FIG. 22 
which are utilized to generate tone correction coefficients, 
from pixel values of corresponding points between the 
images a and b, which are obtained on the basis of Affin 
transformation parameters (A, B, C and D). In the fourth 
modified embodiment, matching processing is performed in 
the neighbor of coordinates of the corresponding points 
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which are obtained on the basis of Affin transformation 
parameters, so that sample pixels are obtained with high 
precision. 

The operation of obtaining sample pixels according to the 
fourth modified embodiment will be described with refer- 5 
ence to FIG. 27. 

In FIG. 27, points P and P are a pair of corresponding 
points obtained on the basis of Affin transformation param- 
eters. Herein, the template T is generated with the point P as 
the center of the image a'. Matching operation is executed 
with respect to points in the template T, within the range of 
the searching objective area S of the image b' having a point 
P' as its center. The obtained corresponding point is defined 
as a point P" which precisely coincides with the point P. 
Accordingly, deviation of sample pixels conventionally gen- 
erated due to errors in the parameters is corrected (an error 
generated between P'-P"), thus more accurate sample pixels 
are generated. 

The result of the matching processing is determined to be 
correct if a value of non-coincidence at a peak value is lower 
than a threshold value. 

Suppose that a matched position cannot be found in the 
searching objective area S, sample pixels may be generated 
by utilizing coordinates of a corresponding point which is 
obtained by Affin transformation. ^ 

The tone correction process of the images a' and b', 
generation process of a combined image c' performed by the 
image combining unit 250 are performed similar to that of 
the second embodiment. 

As set forth above, according to the fourth modified 
embodiment, highly accurate sample pixels can be obtained 
by performing matching processing in the neighbor of the 
coordinates of the corresponding points obtained on the 
basis of Affin transformation parameters. 

The present invention is not limited to the above embodi- 35 
ments and various changes and modifications can be made 
within the spirit and scope of the present invention. 
Therefore, to appraise the public of the scope of the present 
invention, the following claims are made. 

What is claimed is: 

1. An image combining apparatus for combining a first 
image and a second image having an overlapping region 
which overlaps with the first image, comprising: 

identifying means for discriminating the overlapping 
region within the first and second images, and ideati- 45 
fying a first partial image corresponding to the over- 
lapping region of the first image and a second partial 
image corresponding to the overlapping region of the 
second image; 

detecting means for detecting corresponding pixels 
between the first partial image and the second partial 
image, and detecting tone densities of the first and 
second partial images on the basis of detected corre- 
sponding pixels; 

tone correction means for performing tone correction on 
at least one of the corresponding pixels of the first 
partial image and those of the second partial image on 
the basis of a difference in tone density of the first 
partial image and the second partial image so that the 
tone density of the at least one partial image approxi- 
mates to the other; and 

combining means for combining the first image and the 
second image, substituting image data in the overlap- 
ping region with image data indicative of the corrected 
first partial image and the second partial image. 

2. The image combining apparatus according to claim 1, 
wherein said detecting means comprises calculating means 
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for calculating a correction function which prescribes a 
correlation between the tone density of the first partial image 
and the tone density of the second partial image, and said 
tone correction means performs tone correction on either the 
first image or the second image utilizing the calculated 
correction function. 

3. The image combining apparatus according to claim 2, 
wherein the correlation is approximated by a quadratic 
function. 

4. The image combining apparatus according to claim 1, 
wherein said combining means adds weights to at least one 
of the first and second images corrected by said tone 
correction means, adds the weighted image to the original 
image, and combines the added image with the other image. 

5. The image combining apparatus according to claim 1, 
further comprising: 

setting means for setting a boundary at a substantial center 
of the overlapping region, wherein said combining 
means performs correction on pixels of the second 
partial image in correspondence with a distance from 
the boundary to the pixel. 

6. The image combining apparatus according to claim 1, 
wherein said combining means sets a tone conversion area 
in the overlapping region. 

7. The image combining apparatus according to claim 5, 
wherein said combining means performs tone conversion 
only on the second partial image. 

8. The image combining apparatus according to claim 5, 
wherein said combining means performs tone conversion on 
the first partial image and the second partial image. 

9. The image combining apparatus according to claim 5, 
wherein said combining means changes a size of the tone 
conversion area in accordance with a difference in density of 
the first partial image and density of the second partial 

10. The image combining apparatus according to claim 5, 
wherein said combining means sets the tone conversion area 
in accordance with a difference in each average value of the 
first image and the second image within the overlapping 

11. The image combining apparatus according to claim 1, 
wherein said identifying means determines mapping of 
coordinates conversion from a space of the first partial 
image to a space of the second partial image, and 

said combining means utilizes inverse transformation of 
said mapping to map the second partial image into the 
space of the first partial image, thereby combining the 
first image aod the second image. 

12. An image combine method of combining a first image 
and a second image having an overlapping region which 
overlaps with the first image, comprising the steps of: 

discriminating the overlapping region within the first and 
second images to identify a first partial image corre- 
sponding to the overlapping region of the first image 
and a second partial image corresponding to the over- 
lapping region of the second image; 

detecting means for detecting corresponding pixels 
between the first partial image and the second partial 
image, and detecting tone densities of the first and 
second partial images on the basis of detected corre- 
sponding pixels; 

performing tone correction on at least one of the corre- 
sponding pixels of the first partial image and those of 
the second partial image on the basis of a difference in 
tone density of the first partial image and the second 
partial image so that the tone density of the at least one 
partial image approximates to the other, and 
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combining the fiist image and the second image, with 
substitution of image data in the overlapping region 
with image data indicative of the corrected first partial 
image and the second partial image. 

13. The apparatus according to claim 1, wherein said s 
identifying means comprises: 

corresponding-point-extracting means for extracting a set 
of corresponding points from the first image and the 
second image; and 

removing means for removing from the set of correspond- 10 
ing points, data indicative of corresponding points 
which are erroneously detected as corresponding points 
by said corresponding-point-extracting means, on the 
basis of coordinates conversion parameters estimated 
by utilizing the set of corresponding points extracted by 35 
said corresponding-point-extracting means, 

wherein said combining means estimates image combine 
parameters on the basis of the set of corresponding 
points from which erroneous corresponding points 
have been removed. 

14. The image combining apparatus according to claim 
13, wherein said tone correction means extracts pixel values 
of the set of corresponding points, from which the erroneous 
corresponding points have been removed, respectively from 
the first image and the second image on the basis of the 
estimated image combine parameters, and performs tone 
correction respectively on the first and second images uti- 
lizing the extracted pixel values. 

15. The image combining apparatus' according to claim 
13, 30 

wherein said tone correction means generates a first 
correction function for performing tone correction on 
the first image on the basis of a difference between an 
average value of the pixel values extracted from the 3S 
first image and an average value of the pixel values 
extracted from the second image, 

performs tone correction on the first image utilizing the 
fist correction function, 

generates a second correction function to coincide the 40 
pixel values of the corresponding points in the second 
image from which the erroneous corresponding points 
have been removed by said removing means, with the 
pixel values of the corresponding points in the cor- 
rected first image from which the erroneous corre- ^ 
sponding points have been removed, and 

performs tone correction on the second image utilizing the 
second correction function. 
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16. The image combine method according to claim 12, 
wherein said identifying step comprises the steps of: 

extracting a set of corresponding points from the first 
image and the second image; and 

removing, from the set of corresponding points, data 
indicative of corresponding points which are errone- 
ously detected as corresponding points in said extract- 
ing step, on the basis of coordinates conversion param- 
eters estimated by utilizing the set of corresponding 
points extracted in said extracting step, 

wherein in said combining step, image combine param- 
eters are estimated on the basis of the set of corre- 
sponding points from which erroneous corresponding 
points have been removed. 

17. The image combine method according to claim 16, 
wherein said step of performing tone correction comprises 
the steps of: 

extracting pixel values of the set of corresponding points, 
from which the erroneous corresponding points have 
been removed, respectively from the first image and the 
second image on the basis of the estimated image 
combine parameters; and 

performing tone correction respectively on the first and 
second images utilizing the extracted pixel values. 

18. The image combine method according to claim 16, 
wherein said step of performing tone correction comprises 
the steps of: 

generating a first correction function for performing tone 
correction on the first image on the basis of a difference 
between an average value of the pixel values extracted 
from the first image and an average value of the pixel 
values extracted from the second image; 

performing tone correction on the first image utilizing the 
fist correction function, 

generating a second correction function to coincide the 
pixel values of the corresponding points in the second 
image from which the erroneous corresponding points 
have been removed in said removing step, with the 
pixel values of the corresponding points in the cor- 
rected first image from which the erroneous corre- 
sponding points have been removed, and 

performing tone correction on the second image utilizing 
the second correction function. 
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ABSTRACT 



An image processing apparatus is provided with a display 
device to display images on a display screen so that the 
images displayed on the display device can be edited and 
processed for joining of the images. The image processing 
apparatus includes an image specifying device to specify 
two or more images to be joined together on the display 
device. An image arranging device arranges the images 
specified by the image specifying device in such an order as 
to join the images together, and a display control device 
controls the display device to display the images arranged by 
the image arranging device. A joining point specifying 
device specifies any one joining point for each image so that 
vertically or laterally adjacent images displayed by the 
display control device can be joined together by referring to 
the joining points. A joining device joins adjacent images 
together by referring to the joining points specified by the 
joining poiut specifying device. This makes it possible to 
easily join the images together by such a way as to merely 
specify one joining point for each image, and hence to 
execute image joining easily and efficiently. 

63 Claims, 48 Drawing Sheets 
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IMAGE PROCESSING APPARATUS, 
METHOD AND COMPUTER-READABLE 
RECORDING MEDIUM WITH PROGRAM 

RECORDED THEREON, FOR JOINING 
IMAGES TOGETHER BY USING VISIBLE 
JOINING POINTS AND CORRECTING 
IMAGE DISTORTION EASILY 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to image processing appa- 
ratus and method to display images on a display screen, edit 
and process the images displayed, and retrieve the images 
edited and processed, and a computer-readable recording 
medium with a program thereon for making the computer 
execute the method. In particular, the invention relates to a 
technique for joining plural images and correcting distortion 
of the images. 

2. Discussion of the Background 

There has conventionally been known a technique for 
joining plural images together to form an image picture. 
Such a technique would be used in a case where two or more 
images are necessary to photograph a panoramic view, such 
as the entire image of a tall building or an event hall, an 
aerial photograph or a case where the distance from an 
object to be photographed is too near for a normal digital 
still camera to put the entire image in a picture, such as an 
interior photo. In these cases, two or more pictures need to , 
be joined with each other to form the panoramic view. A 
method of joining two images has been known in which two 
images are joined with each other by moving and putting 
either of the images in position by hand while viewing the 
joining part on a screen. 

It is 



related to the observing points, but it is very hard for 
beginners to do these jobs efficiently. When image distortion 
correction needs to be performed for a large number of 
images, detection of observing points and data input of the 
; observing points are required for each image. This makes it 
difficult even for skilled persons to do these jobs efficiently. 

SUMMARY OF THE INVENTION 
^ In order to solve the above conventional problems, an 
object of the present invention is to provide an image 
processing apparatus and method capable of joining plural 
images in an easy operation even for beginners, and a 
computer-readable recording medium with a program 
recorded thereon for making the computer execute the 
' method. 

Another object of the present invention is to provide 
image processing apparatus and method capable of correct- 
ing distortion of images in an easy operation even for 
( beginners, and a computer-readable recording medium with 
a program recorded thereon for making the computer 
execute the method. 



In one aspect of the present invention, an image process- 
ing apparatus is provided with a display device to display 
25 images on a display screen so that the images displayed on 
the display device can be edited and processed for smooth 
joining of the images. The image processing apparatus 
includes an image specifying device to specify two or more 
images to be joined together on the display device. An image 
30 arranging device arranges the images specified by the image 
specifying device in such an order as to join the images 
together, and a display control device controls the display 
device to display the images arranged by the image arrang- 
ing device. A joining point specifying device specifies a 
35 joining point for each image so that vertically or laterally 
known that an image photographed by a digital still adjacent images displayed by the display control device can 
camera or video camera generally contains geometrical be joined together by referring to the joining points. A 
distortion produced by distortion aberration of the lens joining device joins adjacent images together by referring to 
system forming the image in a position deviated from the the joining points specified by the joining point specifying 
original position. To correct distortion of such an image, a 40 device. According to the present invention, the specified 
method is known such as one disclosed in Japanese patent images are arranged in such an order as to be joined together, 
application laid-open publication No. 9-294225, entitled and the arranged images are so displayed that a joining point 
"Method for Deciding Parameter for Image Distortion Cor- can be specified for each image while viewing both images 
rection and Image Pickup Device." This publication dis- displayed adjacent to each other. This makes it possible to 
closes a technique in which, plural images including a 45 easily join the images together by merely specifying one 
common pattern picked up from one point are used to detect joining point for each image, and hence to execute editing 
plural sets of corresponding observing points from these and processing of the images such as image joining easily 
images and measure angles of the observation points with and efficiently. 

respect to an optical axis of the image pickup system, thus In another aspect of the present invention, the image 

based on the 50 processing apparatus further includes a connection line 
drawing device to draw a connection line between joining 



estimating a parameter for distortion 
angle information obtained. 

The above method of joining plural images together to 
form an image picture may be useful for skilled persons who 
are specialized in this work, but it is very hard for beginners 
to do these jobs efficiently. If two or more images are joined 
together to form an image picture, adjacent images need to 
be adjusted one after the other, respectively, and this makes 
it difficult even for skilled persons to do this work efficiently. 
It is also necessary to perform additional image 



points specified by the joining point specifying device, and 
the display control device controls the display device to 
display both the images arranged by the image arranging 
device and the connection line drawn by the connection line 
drawing device. According to the present invention, a con- 
nection line is drawn between the joining points specified, so 
that the operator can both specify the joining points as if he 

~ or she drew a line by hand, and recognize instantaneously 

as to make the joining part inconspicuous. When the images 6 o whether the joining points are specified or not 

? K aD H' CO1 J tcti0D mu f ^ be ^ still another aspect of the present invention, the image 

SSffcS^ *" ln 1 redUCh ° D 1D Passing apparatus, further inLdes a joining-^n, spel 

, ,. fication releasing device to release specification of the 

image distortion correction requires detection of plural previously specified set of joining points when a given set of 
^l^TT 11 ^ 0 ^ s ? rvin S Points from images, the 65 joining points is specified for a pair of adjacent images, 
dostoruon of which should be corrected. It may be possible According to the invention, when a set of joining poinl is 
for skilled persons to detect observing points and input daU specified for a pair of adjacent images for which another set 
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of joining points has already been specified, specification of join images without occurrence of distortion eveo if lie 

the previous set of joining points is released, so that when images have been input by input devices with different focal 

specifying a new set of joining points, the operator can lengths 

ctntfv J C . J edficad0n ° f joinin S Points easily and cfE- In yel of ^ k 

cieallv in the same o Deration as when Knpj-ifuinn a c<-i n f c ... .. v .. F"' 3 *-"' 



joining points for the first time, without the need to release „„ • . - , . . . 

the specification of the previous set of joining points Z *^l & T & apparatus for jo.ning of 

in , ,k . f u - BF mages by performing the above-described steps. This 

In yet another aspect of the present invention, the image makes it possible for a computer to read the computer 

processing apparatus further includes a focal length setting instructions, and hence for the computer to carry out the 

device to set the focal length with which an image has been » met hod for controlling an image processing apparatus for 

input, and an image correcting device to correct the image the joining of images, 

based on the focal length set by the focal length setting i™ criii ™. . f .u 

device. According to ute invention the focal teneth with ^ ° f °? prtStnl lnventl0D . 311 ™ a g e 

which an image has been input is set to beetle imTee S™? 8 ^ 8 * P i° Vlded ^ * device for 

based on the focal lenglh set for foinrngTrLage wifh 15 ^ on a display screen so ma, the images 

another. This makes it possible to join images ^uW t^ortZalttZ^T^ ^ ^ 

distonion, even if the images have been input by input d c lfcT^ * ™age distortion. An image specifying 

devices with different focal lengths. * ^ T ^ m P hot °g ra P h6d 

, , al dl S ntl y different photographmg positions for correcting 

In shll another aspect of the present invention, a method distortion of the images. An image arranging device 

for controlling an image processing apparatus for joining of 20 arranges the images specified by the image specifying 

images includes steps of displaying images on a display device, and a display control device controls the display 

screen, specifying two or more images to be joined together, device to display the images arranged by the image arrang- 

arrangmg the specified unages m such an order as to be ing device. An identical point specifying device specifies an 

joined together, displaying the arranged images, specifying identical point for each image displayed by the display 

a joining point for each image so that adjacent images 25 control device, and a distortion correcting device corrects 

displayed can be joined by referring to the joining points, distortion of images by referring to the identical points 

and joining adjacent images together by referring to the specified by the identical point specifying device According 

specified joining point. According to the invention, the to the invention, image distortion can be corrected by a 

specified images are arranged in such an order as to be simple operation to merely specify one identical point for 

joined together, and the arranged images are so displayed 30 each image displayed, thus editing and processing the 

that a joining pomt can be specified for each image while images easily and efficiently, 

viewing both images displayed adjacent to each other. This i„ „„. „ ,u . <- ... 

makes it possible to join fhe images by a simple operation of proC eS atL"^ for tFT* T^™' f ^ 

specifying one joining point for each image', an hence to ^ n^i^^S^Z^Z^^T,. 

S52iE? rfte " w " i " ,a,t rrr u r hetw r idcntical ^ 

j ;.„ ,, , / . identical pomt specifying device, and the display control 

In still another aspect of the present invention, the method device controls the display device to display both the images 

tor joining of images further includes steps of drawing a arranged by the image arranging device and the connection 

connection line between the specified joining points, and line drawn by the connection line drawing device Accord- 

displaymg not only the arranged images but also the drawn ing to the invention, a connection line is drawn between the 



™...™„n line. According to the invention, a connection identical points specified, so that the operator can specify the 

line is drawn between the joining points specified, so that the identical points as if he or she drew a line by hand and 

operator can specify both the joining points as if he or she recognize instantaneously whether the identical points are 

drew a hne by hand, and recognize instantaneously whether specified or not. 
the joining points are specified or not r t -n , , 

.... , . In still another aspect of the present invention, the image 

In yet another aspect of the present mvention, the method processing apparatus for correction of image distortion 



(-„, „■• r ■ £.!_•,. r™^^ u^a.ams lui i^jiicciiuu oi image aistomon 

tor joining of images, further includes a step of releasing further includes an identical point specification releasing 

specification of the previously specified set of joining points device to release specification of the previously set of 
when a new set of joining points is specified. According to 50 identical points when a new set of identical points is 

the invention, when a set of joining points is specified for a specified for images for which another set of identical points 

pair of adjacent images for which another set of joining has already been specified. According to the invention when 

points has already been specified, specification of the pre- a set of identical points is specified for images for 'which 

vious set of joining points is released, so that when sped- another set of identical points has already been specified 
tying a new set of joining points, the operator can change the 5S specification of the previous set of identical points is 

specification of joining points easily and efficiendy in the released, so that when specifying a new set of identical 

same operation as when specifying a set of joining points for points, the operator can change the specification of identical 

die firs, time, without the need to release the specification of points easily and efficiently in the same operation as when 

■he previous set of joining points. specifying a set of identical points for the first time without 



In yet another aspect of the present invention, the method 60 me nee<1 to release the specification of the previous set of 

for joining of images further includes a step of setting the identical points. 

focal length with which an image has been input, and a step In still another aspect of the present invention the image 

of correcting the image based on the focal length set for use processing apparatus for correction of image disSn 

liT" 18 J r TP T • t n °^ er : Accordin S 10 ,he Either includes a focal length setting device to set a focal 

S™ 1 ,™ focal length with which an image has been cs length with which an image has been input, and the distor- 

mput is set to correct the image based on the focal length set lion correcting device corrects distortion of the image based 

for jommg the image with another. This makes it possible to on the focal length set by the focal length setting device 
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^iS^^^S^^ J""* ^SCRIPnON OF- THE DRAWINGS 

based on the focal length set. This makes it possible to 1 is an illustration functionally showing the general 

perform proper distortion correction of images even if the stmcru re °f an illustrated document creating system includ- 

images have been input by input devices with different focal s mg an mage P rocessin g apparatus according to the present 

lengths. invention commonly used for joining plural images together 

In yet another aspect of the present invention, a method ^"^ff^ ° f * e 

for controlling an image processing apparatus for correction * diagram snowing the hardware configu- 

of image distortion includes steps of displaying images on a raUOD ^ ™ age P rocessin 8 apparatus according to the 

display screen, specifying at least two images photographed io prcs ? t ulventlon commonly used for joining plural images 

at slightly different photographing positions, for correcting g ' and correctiD g distortion of the images; 

distortion of the images, arranging the images specified in FIG. 3 is a functional block diagram showing the structure 

the image specifying step, controlling display of the images ? image processing apparatus according to the present 

arranged in the image arranging step, specifying an identical invention commonly used for joining plural images together 

point for each image displayed in the display control step, « ^ correct ing distortion of the images; 

and correcting distortion of the images by referring to the' FIG - 4 & a functional block diagram showing the arrange- 

identical points specified in the identical point specifying men t of an image altering part and the surroundings in the 

step. According to the invention, image distortion can be ™ a 8 e processing apparatus according to the first embodi- 

corrected by a simple operation to merely specify one ment of present invention for the operation of joining 

identical point for each image displayed, thus editing and M P lural images; 

processing the images easily and efficiently. FIG. 5 is an illustration showing an example of a display 
In yet another aspect of the present invention, the method screen on a display part of the image processing apparatus 
for correction of image distortion further includes a step of according to the first embodiment of the present invention 
drawing a connection line between identical points specified during the operation of joining plural images; 
in the identical point specifying step, and the display control 25 FIG. 6 is an illustration showing another example of the 
step controls display of both the images arranged in the display screen on the display part of the image processing 
image arranging step and the connection line drawn in the apparatus according to the first embodiment of the present 
connection line drawing step. According to the invention, a invention during the operation of joining plural images; 
connection line is drawn between the identical points FIG. 7 is an illustration showing still another example of 
specified, so that the operator can both specify the identical 30 the display screen on the display part of the image process- 
points as if he or she drew a line by hand, and recognize ing apparatus according to the first embodiment of the 
instantaneously whether the identical points are specified or present invention during the operation of joining plural 
n °t- images; 

In still another aspect of the present invention, the method FIG. 8 is an illustration showing yet another example of 
for correction of image distortion further includes a step of the display screen on the display part of the image process- 
releasing specification of the previous set of identical points ing apparatus according to the first embodiment of the 
when a set of identical points is specified for images for present invention during the operation of joining plural 
which another set of identical points has already been images; 

specified. According to the invention, when a set of identical FIG. 9 is an illustration showing yet another example of 
points is specified for images for which another set of the display screen on the display part of the image process- 
identical points has already been specified, specification of ing apparatus according to the first embodiment of the 
the previous set of identical points is released, so that when present invention during the operation of joining plural 
specifying a new set of identical points, the operator can images; 

cteatfv t\t!ttftZ°J- id6QtiCa i ^ T Uy Md 6ffi ; « nGS - l0 °-10c «" Orations for explaining image 

curtly m the same operation as when specifying a set of position replacement executed by an image arranging part of 

IZ^nZnZ^ ^^ t -? e ^ to . Kl6aSC ' he proCCS5ing a P» acc °^g t° firs 

the specification of the previous set of tdentical points. embodiment of the present invention during the operation of 

In still another aspect of the present invention, the image joining plural images; 
CZf Rg t m6th r° d 1°' , d / St ° r l i0n correction 50 FIG. 11 is an illustration showing yet another example of 
mcludes a step of setting a focal length with which an image the display screen on the display part of the image process- 
has been input, and the distortion correcting step corrects ing apparatus according to the first embodiment of the 
distortion of the image based on the focal length set in the present invention during the operation of joining plural 
focal length setting step. According to the invention, the images; " 

tl^ , h3S I 66 ", inpU ' iS 561 10 * nG - Uisaa mustrati0D for ^Plaining an example of a 

correct distortion of the image based on the focal length set. window displayed on the display part of the image proceL- 

ti^TirT * f* P Crfonn r P K djStorti0n C ° rreC - apparatUS according to the" first embodiment' o Z 

^^££££fi%£r input by ^ ES iDVentiOD during * e operation of joining plural 

r J,?!'!! 1 aDOtl ? r aSpe ^-° f &C PrCSeDt invcntion ' a computer 60 FIG.'l3 is a flowchart showing a sequence of processing 

readable recording medium stores computer instructions for steps executed by the image altering part and a display 

controlling an image processing apparatus for image distor- control part according to the first embodiment of the present 

tion correction by performing the above-described steps. invention during the operation of joining plural images; 

Th* makes it possible for a computer to read the instructions FIG. 14 is a flowchart showing processing steps executed 



10/15/2003, EAST Version: 1.04.0000 



US 6,593,938 Bl 

7 8 

FIG. 15 is a functional block diagram showing the embodiment of the present invention during the operation of 

arrangement of an image altering part and the surroundings joining plural images; 

in an image processing apparatus according to the second FIGS. 29a-29d are illustrations showing yet another 

embodiment of the present invention during the operation of example of display on the arrangement switching part and 

joining plural images; s variations of image arrangement in the joining work area of 

FIG. 16 is an illustration showing an example of a display ,ne ima g e processing apparatus according to the fifth 

screen on the display part of the image processing apparatus embodiment of the present invention during the operation of 

according to the second embodiment of the present inven- joining plural images. 

tion during the operation of joining plural images; FIGS. 30a-30f> are illustrations showing yet another 

FIG. 17 is a flowchart showing part of processing steps 10 exam P le of display on the arrangement switching part and 

executed by an image altering part and a display control part variations of image arrangement in the joining work area of 

according to the second embodiment of the present inven- the lmage Processing apparatus according to the fifth 

tion during the operation of joining plural images- embodiment of the present invention during the operation of 

FIG. 18 is a functional block diagram showing the plural images; 

arrangement of an image altering part and the surroundings 15 nGS ' 31a 7 31fc are illustrations showing yet another 

in an image processing apparatus according to the third exam pl e of display on the arrangement switching part and 

embodiment of the present invention during the operation of variations of image arrangement in the joining work area of 

joining plural images; tne ™age processing apparatus according to the fifth 

FIG. 19 is a flowchart showing parts of processing steps embodim i ent ° f ^ present invention during the operation of 

executed by the image altering part and a display control part 20 J °™^ pl ?i al ™ ages; . 

according to the third embodiment of the present invention ,' ^ a 7 32c Uft illustrations showing yet another 

during the operation of joining plural images- example of display on the arrangement switching part and 

FIG. 20 is a functional block diagram 'showing the ^ ariations of ™ a g e arrangement in the joining work area of 

arrangement of an image altering part and the surroundings M T 581,16 apparatus ^cording to the fifth 

in an image processing apparatus according to the fourth , P ™ eDh ° a dU " ng me °P eration ° f 

embodiment of the present invention during L operation of J °™^ ^ lma f S; L 

joining plural images; *' ltj - 33 15 a flowchart showing processing steps of 

FIG. 21 is an illustration showing an example of a display Z^tLt I? TT^L™ T 8 * pm ^^ ^ 

screen on the display part of the image processing apparatus 30 L^ntuTdu^ th™^ e f odunent ° f / he P rese * 

according to the fourth embodiment of the present invention 8 ^ P of joining plural images; 

during the operation of joining plural images- ° 34 "V func,lonal block diagram showing the 

FIG. 22 is a flowchart showing parts of processing steps T^ZZ " ^ """^ ^ ^ """T*^ 

executed by me image altering part and a dismay control P m IboZenfnf ,h § ^"T * 

arr-nrH.-nc L th» f^Ztu ™k„5 ~ . f.u • ■ 1 K embodiment oi the present invention during the operation of 

accorchng to the fourth embodiment of he present mvention 35 correcting of ^ P 

dunng the operation of joining plural images; CT „ ,t . ... . . ° . 

Rin ni. m ;ii,„w. u , c j- , rlG. 35 is an illustration showmg an example of a display 

screen "™ f H.^of ^ r § * ° f * d ^ liy screen on a dis P lav P art of ima g e processing apparatus 

IZcZ to thf fi^ltf T 8 ? ,r CeSS1Dg , aPParatUS acC0rdin S to the ^ ^bodiment of the present invention 

SJrif . embodiment of the present invention durmg me tion of mmc&v distorti £ n of im 

dunng the opera ion o joining plural images; « RG 3fi ^ ^ ^ J * 

dkS^" 1 ^ tratl0Q 1 sh0Wlng f an K° thereXample ° ttbe ^ sct66n ° n ^ <fcpl«y P a " of the image processing 

2,1 H , ^fZr 1 k ^ imag t e f° CeSSing apparatUS aCCOrdm S t0 1116 sixtb embodiment of the present 

E nf ° ,H g thC , fif,h ^ mb . 0dlmei l t 0 thc present invention during the operation of correcting distortion of 

invention during the operation of joining plural images; images; 

FIGS . 25 fl -25c are illustrations showing an example of 45 FIG.37 is an illustration showing still another example of 
£S y .t D arran , semem sw ? tcbin g P«t and variations of the display screen on the display part of the image process- 
image arrangement in a joining work area of the image mg apparatus acc0 rding to the sixth embodiment of the 



fh™^, aPP3ramS r° rdiD K g l ° ^ 6fth T bodiment ° f prescnt invention durin g the °P cration correcting distor- 
the present invention dunng the operation of joining plural jo tion of images; 

™cf,nc i£ -„ . , . FIG. 38 is a flowchart showing a sequence of processing 
MGS.26a-26c are lUustralions showing another example steps executed by the image altering part and a display 
)f disnlav on the arranopmpni cu/ii^hi-nr, now i . r .. . . &r uiu » umfuj 



, » 1 — wAvv.uiwu mi. iujaK& aiicuuji pan ana a aispiav 

of display on the arrangement switching part and variations control part of the image processing apparatus according to 

of .mage arrangement in the joining work area of the image the sixth embodiment of the present invention during the 

processing apparatus accordmg to the fifth embodiment of ss operation of correcting distortion of images- 

the present invention dunng the operation of joining plural FIG. 39 is an illustration showing a resulting image for 

J: ' ,_ which distortion correction has been performed in the image 

f lUi. ^7fl-27c are illustrations showing still another processing apparatus according to the sixth embodiment of 

example of display on the arrangement switching part and the present invention during the operation of correcting 

variations of image arrangement in the joining work area of so distortion of images- 

imbS e C n, P nn h CSSiDg '° ^ ^ FIG. 40 is a flowchart showing processing steps executed 

iSSZ^i^*™^^^,^,* byanimagearrangmgpartofJiinagepLeLingappa. 

PrrY -T« ™a •„ u • raIUS ^cording to the sixth embodiment of the present 

, • 2 f a -^ b are ^lustrations showing yet another invention during the operation of correcting distortion of 

example of display on the arrangement switching part and 65 images; 

variations of image arrangement in the joining work area of FIG. 41 is a functional block diagram showing the 

the image processing apparatus according to the fifth arrangement of an image altering part and the surZdings 
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in an image processing apparatus according to the seventh 
embodiment of the present invention during the operation of 
correcting distortion of images; 

FIG. 42 is an illustration showing an example of a display 
screen on a display part of the image processing apparatus 5 
according to the seventh embodiment of the present inven- 
tion during the operation of correcting distortion of images; 

FIG. 43 is a flowchart showing part of processing steps 
executed by the image altering part and a display control part 
of the image processing apparatus according to the seventh 10 
embodiment of the present invention during the operation of 
correcting distortion of images; 

FIG. 44 is a functional block diagram showing the 
arrangement of an image altering part and the surroundings 
in an image processing apparatus according to the eighth 15 
embodiment of the present invention during the operation of 
correcting distortion of images; 

FIG. 45 is a flowchart showing part of processing steps 
executed by the image altering part and a display control part 
of the image processing apparatus according to the eighth 2 
embodiment of the present invention during the operation of 
correcting distortion of images; 

FIG. 46 is a functional block diagram showing the 
arrangement of an image altering part and the surroundings M 
in an image processing apparatus according to the ninth 
embodiment of the present invention during the operation of 
correcting distortion of images; 

FIG. 47 is an illustration showing an example of a display 
screen on a display part of the image processing apparatus 30 
according to the ninth embodiment of the present invention 
during the operation of correcting distortion of images; and 

FIG. 48 is a flowchart showing part of processing steps 
executed by the image altering part and a display control part 
of the image processing apparatus according to the ninth 35 
embodiment of the present invention during the operation of 
correcting distortion of images. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

Referring to appended drawings, the present invention *° 
will be described in detail below with respect to preferred 
embodiments of an image processing apparatus, an image 
processing method and a computer-readable recording 
medium with a program recorded thereon for making the 4J 
computer execute the method. 

First, description is made to the general structure of an 
illustrated-document creating system including an image 
processing apparatus according to the present invention 
commonly used for joining and joining plural images 50 
together and correcting distortion of the images. 

FIG. 1 is an illustration functionally showing the general 
structure of an illustrated document creating system includ- 
ing an image processing apparatus according to the present 
invention. As functionally shown in FIG. 1, the illustrated- 55 
document creating system is such that a host computer 
system 105 is connected by a network such as LAN to an 
image processing apparatus 100 including an input part 101, 
a processing part 102 and an output part 103. 

The input part 101 performs read processing of images, co 
which may include a scanner, an IC card, a digital still 
camera or a photo CD reader. The input part 101 reads or 
lakes in images, and converts the read images into a prede- 
termined file form before transmission to the processing part 

102 65 

The processing part 102 performs read processing of 
images, such as to register, delete and alter the images. For 



example, a personal computer (PC) may be used for the 
processing part 102. An image data storage part 104, incor- 
porated in the processing part 102 or externally connected to 
the processing part 102 stores processed image data, which 
may include an internal hard disk of the PC, a floppy disk, 
a rewritable CD-ROM, an MO or a DVD. The processing 
part 102 also retrieves image data stored in the image data 
storage part 104, pastes the retrieved image data on another 
document and controls a display to indicate the document or 
controls the output part 103 to print out the document. 

The output part 103 performs the printing processing of 
image data and documents containing the image data to be 
printed out under control of the processing part 102. The 
output part 103 may include a monochrome or full-color 
laser printer or a digital copying machine. The output part 
103 may also include a printer having such a function as to 
print business cards or laminate cards such as IC cards after 
printout of the cards. 

The host computer system 105, connected by a network 
such as LAN to a PC as the processing part 102, includes a 
host computer 106 and plural terminals 107 connected by a 
network to the host computer 106. The host computer 106 
may store therein, for example, a personnel information file 
108 related to personnel affairs inside the company so that 
each terminal 107 can access to the personnel information 
file 108. 

The processing part 102 can be linked with the personnel 
information file 108. For example, image data stored in the 
image data storage part 104 of the processing part 102 can 
be retrieved via the personnel information file 108 by 
inputting personnel information such as an employee num- 
ber from a terminal 107. 

Specifically, the illustrated-document creating system can 
be applied to a business card creating system for creating 
business cards with a photograph of one's face; a postcard 
creating system for creating postcards such as invitation 
cards with illustrated products or photographs of staff's 
faces thereon; an ID card creating system for creating an 
identity card (ID card) with a photograph; a leaflet creating 
system for creating leaflets such as a project document, a 
handbill and a catalog; a card creating system for creating 
illustrated cards; a personnel information system for creating 
and administering a personnel master file with photographs 
of personnel's faces such as personnel's photographs, their 
resumes or self -declarations, a name and address book and 
maps; a used-car search system for searching information 
with photographs of used cars; a slock information planning 
and purchasing system for controlling stock information; a 
system for the civil engineering and construction business, 
which is made up of a combination of a work schedule and 
a construction master file; and construction perspective 
creating system for use in combination with a three- 
dimensional CAD system. These illustrated-document cre- 
atioD systems may use or create color images such as color 
photographs. 

The hardware configuration of the image processing appa- 
ratus 100 is described below. FIG. 2 is a block diagram 
showing the hardware configuration of the image processing 
apparatus 100. Shown in FIG. 2 are a CPU 201 for control- 
ling the entire system, a ROM 202 with a boot program and 
the like stored therein, a RAM 203 used as a work area of 
the CPU 201, an HDD (hard disk drive) 204 for performing 
reading/writing of data with respect to an HD (hard disk) 
205 under control of the CPU 201, an HD 205 for storing 
data written under control of the HDD 204, an FDD (floppy 
disk drive) 206 for performing reading/writing of data with 
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respect to an FD (floppy disk) 207 under control of the CPU The image registration part 302, the image deleting part 

201. a removable FD 207 for storing data written under 303, the image altering part 304, the image retrieving part 

control of the FDD 206, and a display 208 for displaying 305, the image joining part 306, the display control part 307 

documents containing images, function information and the and the printing control part 309 are embodied, respectively, 

s by the CPU 201 or the like executing command processing 
An interface (1/F) 209, connected to a network NET according to commands written in programs such as an OS 

through a communication channel 210 controls an internal and an application program recorded on recording media 

interface with the network NET; a keyboard 211 provided such as the ROM 202, the RAM 203, the hard disk 205 or 

with keys for input of characters, numerical values and me floppy disk 207. 

various instructions; a mouse 212 for moving a cursor and 30 The display part 308 displays documents containing 

designating a selected area and the like; a digital still camera images, character strings and the like under control of the 

213 for photographing images with a CCD; a printer 214 for display control part 307. The display part 308 may be the 

printing out documents; and a bus 215 for connecting the display 208 including a CRT or liquid crystal display, 

above components. The digital still camera 213 may be presem ^ be described first with regard 

replaced by a scanner for optically reading images. 15 , 0 editing accordil)g to an embodiment, where plural 

FIG. 3 is a functional block diagram showing the structure picture images are joined together to form a picture. 
. of the image processing apparatus 100. In FIG. 3, the 

processing part 102 includes, in addition to the image data First Embodiment 
storage part 104, an operation guidance part 301, an image 

registration part 302, an image deleting part 303, an image 20 In ^ embodiment in which plural picture images are 

altering part 304, an image retrieving part 305, an image edited lnto a P icture > the arrangements of the image altering 

joining part 306, a display control part 307, a display part P" 1 304 aod . ^ display control part 307 are first described 

308, a printing control part 309 and the like m more detai1 - FIG - 4ssa functional block diagram showing 

The operation guidance part 301 instructs the image ,he fT 8 ^ ^ ^^W*? 304 ' me 
registration part 302, the image deleting part 303, the image 25 COntro1 part 307 surroundings "» the image process- 
altering part 304, the image retrieving par. 305, the imaje "i™** 0 *" lhe flrS J VZ^T ** ^ 
joining part 306, the display control part 307 and the printing m FI £\ 4 ' ^ ™p al . tenn g P art 304 mc ud " aD . ^ 
control part 309 to operate according to the contents of the ^^.P^ 401 '^^ P art 4 ° 2 > a J 01Iun S 
display part 308. For example, the operation guidance part P ° mt ^J™* P art 403 and a 1°>™* P art 404 
301 includes pointing devices such as the keyboard 211 and 30 Thc . specifying part 401 is for specifying two or 
the mouse 212. more images to be joined together. The image specifying 

The image registration part 302 receives image data P art I 01 s P e f? es ima S es * c operator wants to join in 

transmitted from the input part 101 to register the image data acc ° rdance wrfjan operating instruction from the operation 

as an image data file by adding predetermined data such as glance part 301. 

a file name. The image data file so registered is stored into ^ ima g e arranging part 402 arranges the images speci- 

the image data storage part 104. The image deleting part 303 fied b y the ima e e specifying part 401 in such a layout as to 

deletes any of the image data already stored in the image join them in that order. The images may be arranged in a line 

data storage part 104 from the image data storage part 104 or un6S > vertically or laterally. The images specified by the 

in accordance with a deleting instruction from the operation ima 8 e specifying part 401 are thus arranged or rearranged in 

guidance part 301. accordance with the operating instruction from the operation 

The image altering part 304 adds a change in image data guidance part 301 in such a layout as to join the images in 

of any image data file, already stored in the image data mat or d cr - 

storage part 104 in accordance with an altering instruction The joining point specifying part 403 specifies a joining 

from the operation guidance part 303, and restores the 45 P oml f° r each image so that vertically or laterally adjacent 

altered image data into the image data storage part 104. The images displayed on the display screen of the display part 

image altering part 304 will be described in detail later. 3 "8 under control of the display control part 307 can be 

The image retrieving part 305 retrieves a desired image S oined b y referring to the joining point, 

data file from the image data files already stored in the image The processing contents of the image specifying part 401, 

data storage part 104 in accordance with a retrieving instruc- 50 the image arranging part 402 and the joining point specify- 

tion from the operation guidance part 301. The image ing part 403 will be described in detail later, 

joining part 306 selects image data already stored in the The joining part 404 joins the adjacent images by refer- 

image data storage part 104 so that the selected image data ring to the joining point specified by the joining point 

will be pasted on a document in accordance with an instruc- specifying part 403. Such images can be joined together, for 

tion from the operation guidance part 301. S5 example, by a method such as one disclosed in Japanese 

The display control part 307 controls the display part 308 patent application No. 9-316679 or 10-91125 of the same 

to indicate on its display screen an image of an image data applicant. The image produced by the images joined 

file stored in the image data storage part 104, or a reduced together by the joining part 404 is then assigned a file name, 

image or thumbnail of the image, in accordance with a and stored in the image data storage part 104 based on the 

display instruction from the operation guidance part 301. 60 file name. 

The display control part 307 will be described in detail later. The image specifying part 401, the image arranging part 

The printing control part 309 transmits to the output part 402, the joining point specifying part 403 and the joining 

103 printing data such as image data or an illustrated part 404 are embodied, respectively, by the CPU 201 or the 

document in accordance with a printing instruction from the like executing command processing according to commands 

operation guidance 301. The output part is also controlled id 65 written in programs such as an OS and an application 

accordance with operating instructions from the operation program recorded on recording media such as the ROM 202, 

guidance 301 such as to set the number of prints and the like. the RAM 203, the hard disk 205 or the floppy disk 207. 
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The display control part 307 controls the display part 308 
to display a joining window 700 and to display images 
arranged by the image arranging part 402 in a joining work 
area 701 of the joining window 700, as shown in FIG. 7. 

Next, the operations of the image processing apparatus 5 
and the contents of display screens are described with 
respect to concrete display examples. FIG. 5 is an illustra- 
tion showing an example of a display screen appearing on 
the display part 308 of the image processing apparatus 
according to the first embodiment. Shown in FIG. 5 is the 10 
basic display screen of the image processing apparatus 
according to the first embodiment. In FIG. 5, the display 
screen is made up of three areas: a file name display area 50J 
occupying almost the upper half of the left side of the screen, 
a reduced image display area 502 occupying the lower side 15 
of the file name display area 501, and a selected image 
display area 503 occupying almost the right half of the 
screen. 

Preferably, these areas are not displayed in a multi- 
window form, but are configured as fixed areas for the 20 
purpose of inhibiting the operator from changing the posi- 
tion and size of these areas. Since the screen is displayed in 
the same layout at any time, the operator can grasp these 
areas as one screen. This makes it possible for the operator 
to use the image processing apparatus intuitively without 25 
confusion during the operation and without the need to 
memorize complicated operating procedures, and hence may 
edit and process images efficiently. 

File names stored in the image data storage part 104 such 
as the hard disk 205 or the floppy disk 207 are displayed in 30 
the hierarchical order in the file name display area 501. The 
operator can thus select a desired data file by retrieving and 
specifying the name of the desired data file while viewing 
the file names displayed in the file name display area 501. 

Plural reduced images or thumbnails are created by a 
reduced image creating part scaling down actual image data 
at a predetermined magnification rate, and are displayed in 
the reduced image display area 502. This makes it possible 
for the operator to view the reduced images so as to 4Q 
recognize instantaneously which file the image data corre- 
sponds to. Each file name is also displayed under each 
reduced image. 

Adesired image can also be selected by moving the cursor 
onto the corresponding reduced image and clicking the 45 
button of the mouse 212 or the like, instead of specifying the 
file name in the file name display area 501. In this case, the 
operator can use the reduced image as a clue to the desired 
image data file to achieve easy, high-speed selection of the 
desired image even if the operator has only a vague memory 50 
of the file name. 

If a predetermined key word is registered for each image 
data file, such as "character" or "landscape," "business" or 
"private", the reduced images can be displayed by arranging 
the display sequence of the reduced images based on the key 
words. Since the reduced images are arranged based on the 
key words, a desired image data file can be retrieved at high 
speed even if the number of registered image data files is 
large. 

In the display example of FIG. 5, eight reduced images are 
displayed in each of three rows, i.e., the number of reduced 
images displayed is set to 24, but the arrangement and 
number of reduced images displayed can be varied accord- 
ing to the resolution of the display and the contents of 
displayed images, or the difference in retrieving method. 

The selected image display area 503 is an area for 
displaying an image of the image data file selected by 



specifying the file name in the file name display area 501, or 
by specifying the corresponding thumbnail in the reduced 
image display area 502. The selected images may be dis- 
played in the following modes: a standard display mode for 
displaying the selected image of a standard size, a full- 
screen display mode for scaling up or down the display 
frame of the selected image to display the entire frame of the 
selected image, and a variable magnification display mode 
for displaying the selected image enlarged or reduced to 
predetermined magnification desired by the operator. The 
variable magnification display mode is specified by input- 
ting the percentage of magnification. When the percentage is 
set to 100%, the magnification becomes equal to the size of 
the selected image. When a value smaller than 100% is 
input, the selected image is reduced, and when a value larger 
than 1 00% is input, the image is enlarged. 

By specifying a predetermined command from the menu, 
or double-clicking the image itself, the image currendy 
displayed in tile selected image display area 503 can be fully 
displayed on the display screen while concealing the file 
name display area 501 and the reduced image display area 
502 behind the selected image display area 503. This makes 
it possible to edit and process the selected image more 
efficiently. This full-screen display of the selected image is 
particularly effective in processing a large image. After 
completion of editing and processing operations, the full- 
screen display of the selected image can be resized and 
returned to the home image area by specifying a predeter- 
mined command from the menu, or double-clicking the 
image itself again. The file name display area 501 and the 
reduced image display area 502 are then redisplayed. 

Next, description is made to the operating procedures for 
specifying images executed by the image specifying part 
401. FIG. 6 is an illustration showing another example of the 
display screen on the display part 308 of the image process- 
ing apparatus according to the first embodiment. In FIG. 6, 
the number of reduced images displayed in the reduced 
image display area 502 is set to eight for convenience sake. 

Referring to FIG. 6, plural images to be joined together 
are specified from the reduced images in the reduced image 
display area 502. Although such plural images can be 
specified by using the keyboard to input the file names of 
corresponding reduced images, the plural images are speci- 
fied here by using a pointing device such as the mouse 212 
or the like to move the cursor to the reduced image display 
area and click the button of the mouse 212 or the like. 

Since the plural reduced images need to be specified in 
one specifying operation, the button of the mouse 212 is 
clicked while pressing a specific key such as the shift key on 
the keyboard 211, thus easily specifying plural reduced 

FIG. 6 shows a case where three reduced images 601, 602 
and 603 are specified as images to be joined together. These 
reduced images specified may be made distinguishable from 
the others by displaying the file names of the reduced images 
specified as a negative, or enclosing the reduced images with 
a bold frame, so that the operator can recognize the reduced 
images specified. 

After specifying the reduced images, the operator moves 
the cursor to an "IMAGE CORRECTION" menu 604 on the 
menu bar in FIG. 6, clicks the button of the mouse 212 or the 
like, and further clicks an "IMAGE JOINING" menu 605 
displayed as one of pull-down menus to select the image 
joining function. Then, the joining window 700 is popped up 
and displayed as shown in FIG. 7. 

Next, the processing contents of the image arranging) part 
402 are described. FIGS. 7 to 9 are illustrations showing 
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other examples of display screens on the display part 308 of 
the image processing apparatus according to the first 
embodiment. The joining window 700 includes a joining 
work area 701, a mode switching part 702, an arrangement 
switching part 703, a scale-up button 704, a scale-down . 
button 705, a start button 706, and a cancel button 707. 

Images specified by the image specifying part 401 are 
displayed in the joining work area 701. In FIG. 7, the 
reduced images 601, 602 and 603 specified in FIG. 6 are 
enlarged to predetermined magnification and displayed 
(corresponding to the images 751, 752 and 753, 
respectively). 

The mode switching part 702 has two click modes for the 
pointing device such as the mouse 212 or the like: an image 
position replacing mode 708 and a joining position speci- 
fying mode 709. 1 

Icons are arranged on the arrangement switching part 703 
for switching the arrangement or layout in which the images 
specified by the image specifying part 401 are displayed. If 
three images are to be specified, since the three images will 
be arranged vertically or laterally, the two types of arrange- 2 
ments are shown as icons. As shown in FIG. 8, an icon 710 
indicates a lateral arrangement of three images and an icon 
711 indicates a vertical arrangement of three images. The 
operator can move the cursor onto either of the icons and 
click the button of the mouse 212 or the like to switch the 2 
arrangement of the images displayed on the work area 701. 
The icons displayed on the arrangement switching part 703 
vary in shape and number according to the number of" images 
specified by the image specifying part 401. 

FIG. 7 shows a case where the icon 711 shown in FIG. 8 3 
has been selected. The three images 751, 752 and 753 are 
displayed on the work area 701 in the same arrangement as 
., by aligning these images in the vertical 
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processing currently being performed is stopped, and the 
joining window 700 is closed. 

Further, since joining points need to be specified by the 
joining point specifying part 403 for each pair of the 
adjacent images displayed on the joining work area 701, the 
images must be displayed in such a layout as to join the 
images in that order. For this reason, the image arranging 
part 402 performs position replacement processing of the 

FIGS. 10fl-10c are illustrations for explaining the image 
position replacement executed by the image arranging part 
402 of the image processing apparatus according to the first 
embodiment. At first, the click mode on the mode switching 
part 702 shown in FIG. 7 is switched to the image position 
replacing mode 708. Then, an image to be replaced with 
another is specified. The image specification may be done by 
clicking a desired image, for example, an image A in FIG. 
10a, with the mouse 212 or the like. 

The desired image A is dragged to, and dropped in the 
neighborhood of another image B displayed in a position in 
which the operator wants to replace the image A with the 
image B, as shown in FIG. 10*. As a result, the desired 
image A is moved to the position in which the image B has 
been located, while the image B is moved to the original 
position of the desired image A, as shown in FIG. 10c. 

If the click mode on the mode switching part 702 is in the 
image position replacing mode 708, the same operation can 
be performed repeatedly. This makes it possible for the 
operator to perform image replacement efficiently at high 
speed. 

Next, the processing contents of the joining point speci- 
fying part 403 are described. At first, the click mode on the 
mode switching part 702 is switched to the joining position 



direction. On the other hand, FIG. 8 shows a case where the specifying mode 709. FIG. 11 is an illustration showing 



icon 710 has been selected. Thus, the arrangement of images 
can be changed by such a simple operation as to select the 
icon. 

The scale-up button 704 is to scale up the size of images 
displayed on the joining work area 701. The images are 
scaled up to predetermined magnification each time the 
scale-up button 704 is pressed, i.e., when the operator moves 
the cursor onto the scale-up button 704 and clicks the button 
of the mouse 212 or the like one time. The scaled-up images 
are thus displayed on the joining work area 701, and this 
makes it easy for the operator to specify joining points. 

FIG. 9 is an illustration showing yet another example of 
the display screen on the display part 308 of the image 
processing apparatus according to the first embodiment. 
FIG. 9 shows a case where the scale-up button 704 has been 

The scale-down button 705 is used to scale down the 
images, in a way opposite to the scale-up button 704, to 
predetermined magnification identical to that of the scale-up 
button 704, according to the number of times the button is 
pressed. If the scale-down button 705 is pressed as many 
times as the scale-up button 704 is pressed, the images will 
be returned to the original size. Since the size of images can 
not be scaled down any more in FIG. 7, the scale-down 
button 705 is grayed out, and pressing the button 705 is now 
inhibited. 

The start button'706 instructs the start of joining process- 
ing of the images. Since no joining points have been 
specified yet, the start button 706 is grayed out, and pressing 
the start button 706 is inhibited. 

The cancel button 707 instructs cancellation of the joining 
processing. If the cancel button 707 is pressed, the joining 



-yet 

another example of the display screen on the display part of 
the image processing apparatus according to the first 
embodiment 

In FIG. 11 (or 9), adjacent images in the joining work area 

40 701 are compared by the user to specify, with the pointing 
device such as the mouse 212, respective characteristic 
points at which an identical image character appears on both 
images. To specify the characteristic points accurately, the 
images should be enlarged to a certain extent, for example, 

4S as shown in FIG. 11. 

Specifically, in this example, the tips of lampposts 1102 
and 1103, located on the center left of the overlapped parts 
of both images 752 and 753, are suitable for joining points. 
The joining points are specified by moving the cursor to 

50 these tips 1102 and 1103 and clicking the button of the 
mouse 212 or the like. Coordinate data indicative of the 
joining points specified are then stored in a storage area 
provided in the joining point specifying part 403. Such 
joining points are specified for each pair of vertically or 

55 laterally adjacent images. 

After completion of specifying the joining points for all 
the adjacent images, the operator presses the start button 706 
to start execution of the joining processing. When the 
joining processing is completed, an image resulting from the 

60 joining processing is displayed in the joining work area 701 , 
and a window 1200 is displayed with a message: Is this 
result OK? 

FIG. 12 is an illustration for explaining an example of 
such a window displayed on the display part of the image 
65 processing apparatus according to the first embodiment. In 
FIG. 12, if an "yes" button 1201 is pressed, the image 
resulting from the joining processing is assigned a file name 
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and stored into the image data storage part 104. On the other 
hand, if a "no" button 1202 is pressed, the joining result is 
canceled and the operating procedure returns to the display 
screen just before the start button 706 is pressed, so that 
joining points can be specified again. 

Next, description is made to a sequence of processing 
steps executed by the image altering part 304 and the display 
control part 307 according to the first embodiment. FIG. 13 
is a flowchart showing a sequence of processing steps 

executed by the image altering part and the display control ,„ f , / ,„ n ., ~. , .f ~: r \, 

part according to the first embodiment. In theflowchart of 10 f^iS^ )' ^ J ' . 

V-i ^ .. . „ . . . .... fsten S 1311 1R nftCJfltivM hp. nmn>Hiirp nrw>c tn k a nA vi 



Next, the image arrangement processing executed at step 
S1310 is described in detail. FIG. 14 is a flowchart showing 
processing steps executed by the image arranging part 402 
according to the first embodiment. If joining is instructed in 
step S1302 of the flowchart of FIG. 13 (step S1302 is 
affirmative), determination is made in the flowchart of FIG. 
14 as to whether alteration in image arrangement is 
instructed or not (step S1311). If instructed (step S1311 is 
affirmative), the alteration in image arrangement i: 



FIG. 13, it is first determined whether two 
are specified or not (step S1301). The operation remains 
instep S1301 until two or more images are specified (until 
step S1301 becomes affirmative), and it is then determined 
whether joining is instructed or not (step S1302). 15 

If joining is not instructed in step S1302 (step S1302 is 
negative), it is determined whether other instructions such as 
deletion of images are given or not (step S1303). If no other 
instruction is given here (step S1303 is negative), the 
procedure returns to step S1301, and the subsequent pro- 20 
cessing steps are repeated. If any other instruction is given 
in step S1303 (step S1303 is affirmative), this operation is 
ended after execution of the corresponding processing (step 
S1304). 

If joining is instructed in step S1302 (step S1302 is 25 
affirmative), arrangement processing is executed for the 
specified images (step S1310). The image arrangement 
procedures will be described in detail later. 

It is next determined whether the image arrangement is to 30 
be altered or not (step S1340). If alteration is required (step 
S1340 is affinnative), the procedure returns to step S1310 in 
which the images are rearranged. If alteration is not required 
in step S1340 (step S1340 is negative), it is determined 
whether joining points are specified or not (step S13S0). If 35 
any joining points are specified (step S1350 is affirmative), 
coordinate data of the joining points specified are saved 
(step S1365). The processing steps S1350 through S1370 are 
repeated until all the joining points are specified. When all 
the joining points have been specified (step S1370 is 40 
affirmative), it is further determined whether to alter any of 
the joining points (step S1380). 

If alteration of any joining points is required in step S1380 
(step S1380 is affinnative), the procedure returns to step 
S1350. On the other hand, if alteration of any joining points 45 
is not required in step S1380 (step S1380 is negative), it is 
determined whether the start button is pressed or not (step 
S1381). If the start button is pressed (step S1381 is 
affirmative), joining processing is executed, using the con- 
ventional technique disclosed in Japanese patent application 50 
No. 9-316679 or 10-91125 (which are hereby incorporated 
by reference) (step S1382) to display the joined image on the 
display screen of the display part 308 (step S1383). 

It is next determined whether the displayed result is OK 
or not (step S1384). If the result is not OK (step S1384 is 55 
negative), the procedure returns to step S1350 to specify 
joining points again. On the other hand, if OK in step S1384, 
the result is stored (step S1385), and all the processing is 

If the start button is not pressed in step S1381 (step S1381 60 except the image altering part 304 and the display control 

is negative), it is determined whether alteration in image part 307, the parts common to those of the first embodiment 

arrangement is required or not (step S1390). If alteration is are also not described here. 

required (step S1390 is affirmative), the procedure returns to Next, the image altering part 304 is described. FIG. IS is 

step S1310 and the subsequent processing steps are a functional block diagram showing the arrangement of the 

repeated. On the other hand, if alteration is not required (step 65 image altering part 304 and the surroundings in the image 

S1390 is negative), the should be altered or not. After that, processing apparatus according to the second embodiment 

the subsequent processing steps are repeated. of the present invention. As shown in FIG. 15, the image 



(step S1311 is negative), the procedure goes to the next step 
without execution of any processing. 

It is next determined whether image replacement is 
instructed or not (step S1313). If instructed (step S1313 is 
affirmative), the image replacement is performed (step 
S1314). On the other hand, if not instructed (step S1313 is 
negative), the procedure goes to the next step without 
execution of any processing. 

It is next determined whether an enlargement of the 
images is possible or not (step S1315). If possible (step 
S1315 is affirmative), determination is made as to whether 
the enlargement is instructed or not (step S1316). If 
instructed (step S1316 is affirmative), the enlargement is 
performed (step S1317). On the other hand, if it is not 
possible to enlarge the images (step S1315 is negative), or if 
the enlargement is not instructed (step S1316 is negative), 
the procedure goes to the next step without execution of any 
processing. 

It is next determined whether a reduction of the images is 
possible or not (step S1318). If possible (step S1318 is 
affirmative), determination is made as to whether the reduc- 
tion is instructed or not (step S1319). If instructed (step 
S1319 is affirmative), the reduction is performed (step 
S1320). On the other hand, if it is not possible to reduce the 
images (step S1318 is negative), or if the reduction is not 
instructed (step S1319 is negative), the procedure goes to the 
next step, i.e., step 1340, without execution of any process- 
ing. 

As discussed above, when plural images are joined 
together, the first embodiment makes it easy to specify and 
arrange the images, and to specify joining points for adjacent 



Second Embodiment 

Although in the first embodiment one joining point is 
specified at a point, i.e., by clicking the button of the mouse 
212 or the like, it may be specified using a line, i.e., by 
specifying the joint between adjacent images in a drag-and 
drop operation of the mouse 212 or the like, as discussed in 
the following second embodiment. 

Since the general structure of the illustrated document 
creating system including an image processing apparatus 
according to the second embodiment of the present 
invention, and the hardware configuration of the image 
processing apparatus 100 are substantially the same as those 
of the first embodiment, description thereof is omitted here. 
Further, since the image processing apparatus 100 includes 
substantially the same parts as those of the first embodiment 
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altering part 304 includes an image specifying part 1501, an 
image arranging part 1502, a joining point specifying part 
1503, a joining part 1504 and a connection line drawing part 
1505. Since parts other than the joining point specifying part 
1503 and the connection line drawing part 1505 are sub- 5 
stantially the same as those of the first embodiment descrip- 
tion thereof is omitted. 

The joining point specifying part 1503 has substantially 
the same structure as that of the joining point specifying part 
403 of the first embodiment, in which any one joining point 10 
is specified for each image so that vertically or laterally 
adjacent images, displayed on the display screen of the 
display part 308 under control of the display control part 
307, can be joined by referring lo the joining point, but 
differs from the joining point specifying part 403 in method 15 
of specifying the joining point. The connection line drawing 
part 1505 is to draw a connection line between the joining 
points specified by the joining point specifying part 1503. 
The processing contents of the joining point specifying part 

1503 and the connection line drawing part 1505 will be 20 
described in detail later. 

The image specifying part 1501, the image arranging part 

1502, the joining point specifying part 1503, the joining part 

1504 and the connection line drawing part 1505 are 
embodied, respectively, by the CPU 201 or the like execut- 25 
ing command processing according to commands written in 
programs such as an OS and an application program 
recorded on recording media such as the ROM 202, the 
RAM 203, the hard disk 205 or the floppy disk 207. 

The display control part 307 controls the display part 308 
to display not only images arranged by the image arranging 
part 1502, but also a connection line 1621 drawn by the 
connection line drawing part 1505 as shown in FIG. 16. 

Next, description is made to specification processing of 35 
joining points and drawing processing of a connection line 
executed by the joining point specifying part 1503 and the 
connection line drawing part 1505. FIG. 16 is an illustration 
showing an example of a display screen on the display part 
308 of the image processing apparatus according to the 40 
second embodiment. In FIG. 16, the operator switches the 
click mode to the joining position specifying mode, com- 
pares adjacent images 1602 and 1603 to specify character- 
istic points at which an identical image character appears on 
both images. The operator then moves the cursor to one 45 
point 1612 on the image 1602, clicks (presses) the button of 
the mouse 212 or the like, and moves (drags) the cursor to 
the other point 1613 on the image 1603 by moving the 
mouse 212 while maintaining a pressed state of the button. 
The connection line drawing part 1505 is synchronized with 50 
the movement of the cursor to draw the connection line 1621 
on the coordinates identical to the cursor path on the screen. 
When the cursor reaches the point 1613, the operator 
releases (drops) the pressed state of the button. The connec- 
tion line 1621 drawn by the connection line drawing part 5S 

1505 is thus fixed. 

The joining points are specified as such above. Coordinate 
data of the joining points specified are saved (stored) in a 
storage provided inside the joining point specifying part 

1503. As in the first embodiment, the specification process- 60 
ing of joining points is performed for each pair of vertically 

or laterally adjacent images. After completion of the speci- 
fication processing for all the adjacent images, the operator 
can press the start button to start execution of joining 
processing. ( 

Thus, the joining points can be specified by the simplest 
way by pressing the button of the mouse or the like, moving 
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the mouse while pressing the button, and releasing the 
pressed button in a predetermined position. 

The connection line 1621 may be such a colored line that 
it can be easily discriminated, such as a red line. The 
connection line may also have such a thickness that is can be 
easily discriminated The line types, such as color and 
thickness, may be selectable by the operator, or selectable 
automatically depending on the image by taking into 
account a color used in the image so that the connection line 
can be more easily discriminated from the color. 

The connection line 1621 may be displayed as a dotted 
line during drag operation, and changed to a solid line after 
drop operation to fix the connection line. This makes it 
possible for the operator to easily confirm the process of 
specifying the connection line. 

Next, description is made to a sequence of processing 
steps executed by the image altering part 304 and the display 
control part 307 according to the second embodiment FIG. 
17 is a flowchart showing part of processing steps executed 
by the image altering part 304 and the display control part 
307 according to the second embodiment. Since the flow- 
chart of FIG. 17 executes the same processing steps as those 
from S1301 to S1340 and those from S1365 to S1390 in the 
flowchart of FIG. 13 according to the first embodiment, the 
common steps and their description are omitted. 

In step S1340 of the flowchart of FIG. 13 according to the 
first embodiment, if alteration in image arrangement is not 
required (step S1340 is negative), determination is made in 
the flowchart of FIG. 17 as to whether the button of the 
mouse 212 or the like is pressed or not (step S1351). If the 
button is pressed (step S1351 is affirmative), it is determined 
whether the cursor is positioned on an image or not (step 
S1352). If the cursor is not positioned on any image (step 
S1352 is negative), an error indication is given (step S1353) 
and the procedure returns to step S1351. On the other hand, 
if the cursor is positioned on an image (step S1353 is 
affirmative), drawing of a connection line is started at the 
point (step S1354). 

After that, the operation remains in step S1355 until the 
button of the mouse 212 or the like is released. When the 
button is released (step S1355 becomes affirmative), it is 
determined whether or not the cursor is positioned on an 
image adjacent to the image on which the cursor has been 
located in step S1352 (step S1356). If the cursor is not 
positioned on the adjacent image (step S1356 is negative), 
an error indication is given (step S1353), and the procedure 
returns to step S1351 to repeat the subsequent processing 

If it is determined in step S1356 that the cursor is 
positioned on the adjacent image (step S1356 is affirmative), 
the drawing of the connection line is ended (step S1361). 
After that, the procedure goes to step S1365 of FIG. 13 
according to the first embodiment. 

As discussed above, according to the second embodiment, 
a connection line is drawn between joining points specified, 
so that the operator can not only specify the joining points 
as if he or she drew a line by hand, but also recognize 
instantaneously whether the joining points are specified or 



Third Embodiment 



The ! 



of joining points already specified may be 
released when a new set of joining points is specified for a 
pair of adjacent images, as described in the following third 
embodiment. Since the general structure of the illustrated 
document creating system including an image processing 
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apparatus according to the third embodiment of the present S 13 65 to S1390 in the flowchart of FIG 13 according to the 

invention, and the hardware configuration of the image first embodiment, the common steps and their description 

processing apparatus 100 are the same as those of the first are omitted. Further, the flowchart of FIG. 19 also executes 

embodiment, description thereof is omitted here. Further, the same processing steps as those from S1351 to S1356 and 

since the image processing apparatus 100 includes the same s S1365; description thereof is omitted as well 

th0 , Se ™°/ £° firS ! 6mbodimeal e «*P' * e In step S1356, if the cursor is positioned on the adjacent 

altering part 304, the parts common to those of the first imape • , Km , t; JL\ ; ■ ,w ^ c * . . 

rmhnHi'mrnt „„. j«„-l„. l„. image (step is amrmative), it is then determined 

embodiment are also not described here. whelher joining poinU m ^ fof ^ a 

Next, the image altenng part 304 is described. FIG. 18 is or not (step S1357). If the joining points are not specified 

a functional block diagram showing the arrangement of the 10 (step S1357 is negative), the procedure shifts to step S1361 

image altenng part 304 and the surroundings in the image without execution of any processing 

processing apparatus according to the third embodiment of r>„ „,i,„ i, a -e.u ■ ■ ■ • r. . . 

the present invention. As shown in FIG. 18. the image Sl %t S ^Y**? 0 }™* ^ are s Pf ^ sle P 

altenng part 304 includes an image specifying part 1801, an ™ ^ \ ^"T P °T ^ ^H**? ^ P 

1803, a joining part 1804, a connection line drawing part l0 step S1361 v>y>^*i* sues 

1805 and a joining-point specification releasing part 1806 A _,. 

Since parts other than the joining point specifying part wh ^ ^ T^ 8 ^ '«» *W_embodiment, 

1803, the connection line drawing part 1805 and me joining- 7^1 h.Vh P ,°T " SP f °L ad J accnt ' ma S 6S 

point specification releasing part 1806 are the same as thoie f ° r W J"f an ° ther Set of J olnm f P° lnts has alread y been 

of the first embodiment, and the connection line drawing L 3 ° lma ^, m \ ^edification releasing part 1806 

part 1805 is the same as that of the second embodiment "T? P KVl " ° f J°"" n S P™* already specified, 

description thereof is omitted ' so that when SP 60 ^ a new set of joining points, the 

^^^^^^^ * sss^ss^^ss^ 

pining point specifying part 403 of thefet embodiment, or 25 of joinmg points for ^ &st ^ ^ £ ™ 

& joining point specifying part 1503. The joimng-point the specification of the previous set of joining points, 
specification releasing part 1806 releases specification of the 

previous set of joining points when a set of joining points is Fourth Embodiment 

specified for a pair of adjacent images for which another set Alihmmh th* a h™,„ fi~i . .k a u a- t. 

of joining points has already been specified. The processing 30 f ^ ^ ab ° ve , firsl '° third embodiments have been 

contents of the joining-point specification releasing part ""T P*f>S' a P hed ™. th a ^ ^ c ^era 

1806 will be described in detail later. 2 ° ha T g . fi f d ^ 1jC - theSE embodiments 
7l • ., . . 1 . assume that the focal length is invariable, the focal length 

180? T ffn P6C1 ^ SPar T W ' ^^Z\ ge (1 ! rr • angmgPart ^ be Xt for P»per joining of images even if the imafes 

1802, Rejoining point specifying part 1803. the joining part 3f have been photographed whh a digital still camera 213 

1804 the connection line drawing part 1805 and the joining having variable fo B ca f lengthS) „ ^ the {o ™J™ 

pomt specification releasmg part 1806 are embodied, fourth embodiment "mowing 

respectively, by the CPU 201 or the like executing command wi,,„ t • * ... 

processing according to commands written in programs such , n(r „„ h L ? a*^*. T*^ l° l T g " C ph °" 

as an OS and an application program recorded onrecording , n Stilt l , ^' T, ^ ^ 

media such as the ROM 202, the RAM 203, the hard disk 40 J*T ,™ de P endln g on _ the focal lcn 8 th > * is 

205 or the floppy disk 207 difficult to join these images together. The fourth embodi- 

\wt •, „ a < i " • , * ment Qas b6e n made to solve the problem by standardizing 

Next, descnption is made to releasing processing of a set the focal len ^ h of ^ ; J , hat ^ \ * 

re/eaX 8 P « 1 ^ S P ecificat i on joined together. Since the general struciire o Mi^trated 

n rlZT.r ^^ 806 -^ enas f o f .J 0 ^gP°^«^cmed 45 document creating system g including ^ i ocessin ^ 

SeLw nLf^n. ima86S ' ^ } °T g ~ P T S P £Clfi f Catl0n apparatus accordhTg to the fourth embodiment of L present 

n? f T S K wheth 5 ™ other k set ° f J 0 " 1 " invention, and the* hardware configuration of the^age 

Further, when a new set of joining points is specified, the altering part 304, the parts common to those of the first 

joimng-point specification releasing part 1806 releases the embodiment are also not described here 
nate" daTa 3 fflS, f 3 . 11 ^ 0 ^, N-t, the image altering part 304 is described. FIG. 20 is 

tH,, • • • , ' , 3 , • B , miage altenng part 304 and the surroundings in the image 

Next, descnpt,on is made to a sequence of processing eo image arranging part 2002, adjoining point specifying part 

steps executed by the image altenng par, 304 and the display 2003, a joining part 2004 a coiSion hne'dXS P- 

Z T P I I I ^ ^ f ^ embodunenl - nG - 19 2005, a joining-point specification releasing part 2006 a 

L fZf T h0W,08 t P^ of Procejmgstepsexeaited by focal length setting part 2007 and an image Meeting par^ 

the image altenng part 304 and the display control part 307 2008. The parte other than the focal length setting parT2007 

Sint^h fl H ^"TZTZ u 65 ^ d '°e image correcting part 2008 arefhe same^thoS Vol 

Since the flowchart of HG 19 executes the same pro- the first to third embodiments, and therefore description 

cessing steps as those from S1301 to S1340 and those from thereof is omitted. 
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The focal length setting part 2007 sets the focal length of setting of the focal length may be added to the image data 
an .mage, photographed with a digital stdl camera 213 or the In this case, the focal length may be automatically set by 
like in accordance with an operating instruction from the reading the information related to setting of the focal lenelh 

tk • ' " close ,he focaJ Ien S ,h setting window 2102. 

The image correcting part 2008 corrects the image to be " Ne „. H<*a-rintir,n ,c ™*a . c 



^s^s^r",^^- 11 " 15 s^ot^s^ ^ con,rcl pari 
sss sls, t a as X" sis s " F ^„ 22 — - the — 

distortion aberration coefiScient. fine distortion may be a™ P" 1 ^ 13 , 10 fr ° m 

corrected, for example, by a method such as one disclosed f 3 ™ '° m ' he & °T^ n ° f F1G U aCCOrdin S to «« 

in Japanese application No. 9-303893 (U.S. patent applica- 20 ^ embodiment . the marks of the common steps and their 
tion Ser. No. 08/807,571, filed Feb. 27, 1997) of the same descrrotl °° m onu,ted - The flowchart of FIG. 22 also 
applicant executes the same processing steps as those from S1311 to 

2oST ag ^ edfy 7 p n 200i '^ ageamngingpwi ^?^^-^^^^ d fi i 

2002, the joining point specifying part 2003, the joining part we n P as 

2004, the connection line drawing part 2005, the joining- 25 ,„ lh „ u ^ c . c 

point specification releasing part 2006, the focal lengfh J? c,™ 1 ^ ^ * ^ S1319 * DegatiVe ° f 

setting part 2007 and the image correcting part 2008 Le w S th 11 T' dcleim ™ h0D . 15 made as to 

embodied, respectively, by the CPU 201 or the like execut- Sett ^? P*™**"* 15 instructed or not 

ing command processing according to commands written in £ f}^ " (step 513311 is affirmative), the 

programs such as an OS and an application program 30 ^"^fh ««ing Processing is performed (step S1332). 

recorded on recording media such as the ROM 202 the J* °^ ^ lf T mStniCted ste P S1331 " ne S alive )> 

RAM 203, the hard disk 205 or the floppy disk 207. e^cS^a^ro ^ ^ S134 °' 

Next, description is made to focal-length setting process- ° ^ P rocessm S- 

ing executed by the focal length setting part 2007. FIG 21 ,y, ^ dl ^ ussed above ' according to the fourth embodiment, 

is an illustration showing an example of a display screen on 35 ^ length f" 1 " 6 part 2207 56,5 the focal ^ 

the display part 308 of the image processing apparatus t^S^' 6 * . T' ^ T agC correctin 6 

according to the fourth embodiment. There is shown in FIG T 2208 °° ^ T g6 baSCd ° D f ° Cal lcngth set ' 

21 a focal length button 2101 to open a focal length settine * JOmmg the m ! ,ge ^ aDOther nis makes il P ossible to 

window 2102 for setting the focal length In FIG 21 sinci J oinima g cs together without occurrence of distortion even if 

the focal length setting button 2101 has already' been 40 f ™ a S<* have been ln P ut b V input devices with different 

pressed, the focal length setting window 2102 is in the lengths. 

pop P ed-up state. Fjfth Embodiment 

The focal length setting window 2102 includes an input c - ... „ , . L J . 

frame 2103 into which the type of digital still camera 213 is u C , t0 foUrth embod ™ents assume a case 

input, a zoom switching part 2104, an OK button 2105 and 45 Z ^f™ 1 ™. of ^ ™ a S es can be arranged both in 

a cancel button 2106. vertlcal direction and in the lateral direction, such a 

n, P n™. n f,^-,.i .n « . . , . number of images (two, three, four, six or nine) as to shape 

button is pressed to display 'a list of the names of dffiS so 1^ ^v^UyT^rX cante' °J i'SSLT^ 

cameras is stored ^ 111 Wldtii ) can bc arranged. This embodiment 

t, ' , . . K dlscu ssed below on the assumption that such a number of 

rel^H fnT T P f 2104 k 'u SWltCh ^ mode 55 ™ a g« « to make the joined image into a rectangular shape 

dated to the zoom depending on whether the image to be can be set, but the present invention is not limited by these 

Zen no'zcTm ST 1 " w ^ ^ ^ ^ *«* - mbina ^- of ^ages as to form unique 
When no zoom « used for the photograph, the mode is shapes other than rectangles may be set. Further, plural 

35 Smef A ; l e ;'„ the h S ^ dar f f °f 1Cnglh f ° r 1,16 11113865 SUCh 35 four ^ Ave iLges, six imaged, seven 
for „h^ P I r hMd, ^ eD ^ eM0ID ^ USed 60 ™ ages ' ^ ™«B«. ™ages so on may be » 
for the photograph, the mode is swtched to "maximum," joined that a panorama of landscape type can be formed 

AlThouTh^nThT f0f h Ule f1 Caille t ra Spe t ed ' • SiQCC lhe ^ of me Sustrated document 

berwl^n £, m™S f emb ? dlme,lt sw,tch,n 8 15 enabl&d ""tog system including an image processing apparatus 

cT™J^fo?tZLZ^l' m ° re <h Z lWO jr m , m ° dai aCC ° rdiD 5 t° «« fif«> embodiment of L present inVenUon, 

fh?,vnT^i or , swlt , < r h 1 ln S over arnang them depending on 65 and the hardware configuration of the miage processins 

^ml^JSf , T ameraS ° f 10 a PP aratus 100 «* ^ same 45 tbo * of the firsf enSSm 

some types of digital stdl cameras, the mformation related to description thereof is omitted here. Further, the furSal 
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configuration of the image processing apparatus 100 is the 
same as that of the first to fourth embodiments and therefore 
description thereof is omitted as well. 

FIG. 23 is an illustration showing an example of a display 
screen on the display part 308 of the image processing 
apparatus according to the fifth embodiment. A joining 
window 2300 includes a joining work area 2301, a mode 
switching part 2302, an arrangement switching part 2303, a 
zoom switching part 2304, a camera switching part 2305, a 
scale-up button 2306, a scale-down button 2307, a start 
button 2308 and a cancel button 2309. 

As in the joining work area 701 of the first embodiment, 
images specified by the image specifying part 401 are 
displayed in the joining work area 2301. In FIG. 23, five 
images (2351, 2352, 2353, 2354 and 2355) specified from 
corresponding reduced images or thumbnails are displayed 
at a predetermined magnification rate. 

As with the mode switching part 702 of the first 
embodiment, an image position replacing mode 2311 and a 
joining position specifying mode 2312 are displayed on the 
mode switching part 2303 as click modes of the pointing 
device such as the mouse 212. Since the image position 
replacing mode 2311 and the joining position specifying 
mode 2312 have the same contents as the image position 
replacing mode 708 and the joining position specifying 
mode 709, description thereof is omitted here. 

Arrangement alternatives selectable as display layouts of 
images specified by the image specifying part 401 are 
displayed on the arrangement switching part 2303. The 
arrangement switching part 2303 will be described in detail 
later. 

Selectable zoom switching alternatives related to whether 
an image to be joined with another has been photographed 
in the zoom mode or not are displayed on the zoom 
switching part 2304. When no zoom is used for the 
photograph, the mode is switched to "standard" 2313. On " 
the other hand, when the zoom is used for the photograph, 
the mode is switched to "maximum" 2314. The zoom mode 
is thus switched. 
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images, in a way opposite to the scale-up button 2306, to 
predetermined magnification identical to that of the scale-up 
button 2306, according to the number of times the button is 
•pressed. If the scale-down button 2307 is pressed as many 
s times as the scale-up button 2306 is pressed, the images will 
be returned to the original size. 

The start button 2308 is similar to the start button 706 of 
the first embodiment, to instruct the start of joining process- 
ing of the images. Since no joining points have been 
10 specified yet, the start button 2308 is grayed out, and 
pressing the start button 2308 is inhibited. 

The cancel button 2309 is similar to the cancel button 707 
of the first embodiment, to instruct cancellation of the 
joining processing. If the cancel button 2309 is pressed, the 
joining processing currently being performed is stopped, and 
the joining window 2300 is closed. 

Further, since joining points need to be specified by the 
joining point specifying part 403 for each pair of the 
2q adjacent images displayed on the joining work area 2301, 
. the images must be displayed in such a layout as to join the 
images in that order. For this reason, the image arranging 
part 402 performs position replacement processing of the 
images. The position replacement processing of the images 
^ is performed in the same manner as that in the first embodi- 
ment and therefore description thereof is omitted here. 

FIG. 24 is an illustration showing another example of the 
display screen on the display part 308 of the image process- 
ing apparatus according to the fifth embodiment. In FIG. 24, 
l0 the click mode on the mode switching part 2302 is switched 
from the image position replacing mode 2311 to the joining 
position specifying mode 2312. Then adjacent images in the 
joining work area 2301 are compared to specify, with the 
pointing device such as the mouse 212, respective charac- 
5 teristic points at which an identical image character appears 
on both images. FIG. 24 shows a case where connection 



lines 2361, 2362, 2363 and 2364 are drawn as the specifi- 
cation results of the characteristic points. For example, the 
connection line 2361 connects the characteristic points of 
Although in this embodiment, switching is enabled by 4Q the bow appearing on both images 2351 and 2352 
selecting one switching alternative out of two kinds of When one image differs in brightness from another, the 
than two zoom modes may be brightest image is used to standardize the brightness of the 
other images thereto. Alternatively, the leftmost image may 
be used to standardize the brightness of the other images, or 
4 5 any image selected by the operator may be used to stan- 
dardize the brightness of the other images. 

Since the connection lines 2361, 2362, 2363 and 2364 are 
drawn in the same manner as the connection line 1621 is 
drawn in the second embodiment, description of the drawing 
50 method is omitted here. Further, the image joining method 
is also the same as that in the first embodiment and therefore 



switching alternatives, 

used for switching, depending on the types of digital still 
cameras and the kinds of zooms. In some types of digital still 
cameras, the information related to setting of the focal length 
may be added to the image data. In this case, the focal length 
may be automatically set by reading the information related 
to setting of the focal length. 

A list of the names of selectable digital still cameras is 
displayed on the camera switching part 2305. The operator 
can select, out of the listed names, the name of a digital still 
camera with which an image to be joined with other images 
has been photographed. The camera switching is thus per- 
formed. 

As with the scale-up button 704 of the first embodiment, 
the scale-up button 2306 is a button to enlarge the size of 
images displayed on the joining work area 2301. The images 



description thereof is omitted as well. After completion of 
drawing all the connection lines, the start button 2308 enters 
a state in which the operator can press it. Pressing the start 
button 2308 allows the start of image joining processing. 

Next, arrangement switching processing executed by the 
arrangement switching part 2303 is described. FIGS. 25 



are enlarged to predetermined magnification each time the through 32 are illustrations showing examples of displayed 

scale-up button is pressed i.e., when the operator moves the modes on the arrangement switching part 2303 of the image 

cursor onto the scale-up button 2306 to click the button of 60 processing apparatus according to the fifth embodiment, and 

the mouse 212 or the luce one time^The images are thus arrangements or layouts of images in the joining work area 

enlarged on the joining work area 701 at a predetermined 2301.InnGS.25 fl through32 a showexamplesofdispla ye d 

™STr? , rate - ^P 1 ^ on lhe j oi "ing work area modes on the arrangement switching part 2303, and FIGS 

2301. This makes it easy for the operator to specify joining 25fr-32fc, and FIGS. 25c-32c, show variations of image 

pomts - 65 arrangements in the joining work area 2301. 

The scale-down button 2307 is a button, a ' " " 



. is similar to 

scale-down button 705 of the first embodiment, to reduce the 



FIG. 25a shows displayed modes o 
switching part 2303 when two images ai 



the arrangement 
; specified. When 
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two images are specified, there are two arrangement modes, 
a mode for arranging the two images laterally (2 lalerallyxl 
vertically), and a mode for arranging the two images verti- 
cally (1 laterallyx2 vertically). FIG. 2Sb shows an image 
arrangement in the joining work area 2301 when one 
arrangement alternative "2 laterallyxl vertically" is speci- 
fied. FIG. 25c shows another image arrangement in the 
joining work area 2301 when the other arrangement alter- 
native " 1 laterallyx2 vertically" is specified. 

FIG. 26 shows a case where three images 
FIG. 27 shows a case where four images are specified, FIG. 
28 shows a case where five images are specified, FIG. 29 
shows a case where six images are specified, FIG. 30 shows 
a case where seven images are specified, FIG. 31 shows a 
case where eight images are specified and FIG. 32 shows a 
case where nine images are specified. 

For landscape type images in FIGS. 27a to 32a, the word 
"panorama" is specially added so that the operator can see 
it at a glance. Further, since in the embodiment the image 
arrangement is decided based on the case where a maximum 
of three images can be arranged both in the vertical direction 
and in the lateral direction, arrangement alternatives for 
arranging four or more images in the vertical direction or in 
the lateral direction are not displayed except in respective 
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Herein below, description will be made to image editing 
related to correction of image distortion realized by the 
present invention. Since the general structure of the 
illustrated-document creating system including the image 
processing apparatus and the hardware configuration of the 
image processing apparatus 100 for correcting image dis- 
tortion are substantially the same as those used in the - 
above-described embodiments for joining plural images 
together to form an image picture, description thereof is 
specified, i° omitted here. Further, since parts in the image processing 
" ' apparatus 100 other than the image altering part 304 and the 

display control part 307 are substantially the same as those 
used in the above-described embodiments for joining plural 
images together to form an image picture, description 
15 thereof is omitted as well. 

Sixth Embodiment 
First, the arrangements of the image altering part 304 and 
the display control part 307 are described in more detail. 
20 FIG. 34 is a functional block diagram showing the arrange- 
ments of the image altering part 304, the display control part 
307 and the surroundings in the image processing apparatus 
according to the present invention for correcting image 
distortion. As shown in FIG. 34, the image altering part 304 
cases of panorama. For example, when eight images are 25 includes an image specifying part 2401, an image arranging 
specified, arrangement alternatives "4 laterallyx2 vertically" 2402, an identical point specifying part 2403 and a 

and "2 laterallyx4 vertically" are not displayed in FIG. 31a. distortion correcting part 2404. 

Such arrangement alternatives, however, may be displayed At least two images, which arc photographed by slightly 
as required. changing the photographing position and for which the 

Next, camera switching processing executed by the cam- operator desires to correct distortion, are specified by the 
era switching part 2305 is described. FIG. 33 is a flowchart image specifying part 2401. The image specifying part 2401 
showing processing steps of switching the camera mode in specifies images, for which the operator desires to correct 
the image processing apparatus according to the fifth distortion, in accordance with an operating instruction from 
embodiment. In the flowchart of FIG. 33, the number of ,ne operation guidance part 301. 

pixels for an image is first acquired from the image data The image arranging part 2402 arranges the images 
storage part 104 (step S3301). It is then determined, from the specified by the image specifying part 2401 . At least w 



acquired information on the number of pixels, whether a 
digital still camera or cameras different in the number of 
pixels from the photographed image exist in a list or not 
(step S3302). 4 

If any mismatched digital still cameras do not exist in step 
S3302 (step S3302 is negative), the procedure shifts to step 
S3304 in which all the digital still cameras previously 



images photographed by slightly changing the photograph- 
ing position are arranged vertically or laterally so that the 
identical point specifying part 2403 can easily specify an 
identical point on adjacent images. 

The identical point specifying part 2403 specifies any one 
identical point for each image displayed on the display 
screen of the display part 308 under control of the display 
registered in the list of digital still cameras are displayed. On 45 control part 307. The identical point specifying part 2403 



the other hand, if a mismatched digital still 
cameras exist (step S3302 is affirmative), all the names of 
the mismatched digital still cameras are deleted from the list 
(step S3303), and the updated list in which all the mis- 
matched camera names have been deleted is displayed (step 
S3304). 

As discussed above, according to the fifth embodiment, 
when the operator wants to take a picture of a landscape type 
scene, he or she has only to photograph plural images while 



specifies, in accordance with an operating instruction from 
the operation guidance part 301, an identical point the 
operator desires to specify. 

The processing contents of the image specifying part 
2401, the image arranging part 2402 and the identical point 
specifying part 2403 will be described later. 

The distortion correcting part 2404 corrects distortion of 
an image by referring to the identical point specified by the 
identical position specifying part 2403. Images photo- 



changing the position of the digital still camera slightly in 55 graphed by an image pick-up device such as a digital still 

the lateral direction so that the plural images can be joined camera or video camera may include geometrical distortion 

together to form a panorama image. due to image formation in a position deviated from the 

Further, since proper arrangement alternatives arc dis- original position of the image under the influence of 

played based on the number of photographed images, the so-called distortion aberration of the lens system. To correct 

image arrangement can be performed efficiently. 60 image distortion, a method such as, for example, one 

Furthermore, since the number of pixels for an image described in Japanese patent application laid-open publica- 

photographed is acquired and only the digital still cameras lion No. 9-294225 (Japanese patent application No. 

capable of photographing an image corresponding to the 8-2732949) of the same applicant, can be used, 

number of pixels are displayed, erroneous selection of a The method is characterized in that plural images includ- 

digital still camera different from the digital still camera 65 ing an identical point picked up from one point are used to 

used for photographing the corresponding image can be detect plural sets of corresponding observing points from 

avoided. mese images M d measure angles of the observation points 
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with respect to an optical axis of the image pick-up system, as described in the embodiments for joining phira] images 

thus estimating a parameter for correcting distortion of the together to form an image picture, description thereof is 

images based on the angle information obtained. omitted here. 

A file name is assigned to the image of which the The start button 706 instructs the start of correction of 

distortion has been corrected by the distortion correcting 5 image distortion. In the state illustrated in FIG. 36, since 

part 2404 and the image is stored into the image data storage joining points have not been specified yet, the start button 

part 104 based on the file name. 706 is grayed out. and pressing the start button 706 is 

The image specifying part 2401, the image arranging part inhibited. 

2402, the identical point specifying part 2403 and the V 1 * canccl button 707 instructs cancellation of the dis- 

distortion correcting part 2404 are embodied, respectively io tortio11 correction processing. If the cancel button 707 is 

by the CPU 201 or the like executing command processing P K f ed > * e distortion correction processing currendy being 

according to commands written in programs such as an OS " Sam™ I B ^PP" 1 ' ^ ^ distortion correcting window 

and an application program recorded on recording media ?? 15 c } osed - 

such as the ROM 202, the RAM 203, the hard disk 205 or • .', descn P tlon } & mad e to the processing contents of the 

the floppy disk 207 15 ldentlcal P°mt specifying part 2403. FIG. 37 is an ilhistra- 

correcting window 700. device such as the mouse 212, an identrcal point appearing 

T. , , . , p * ,„„ . m each of the adjacent images, i.e., a characteristic point in 

tiaS ti?£Z h the h dls P 1 1 ay P 411 308 15 subs,an - * common image pattern appearing oil both images. To 

tially the same as the case where plural .mages are joined specify the characteristic identical point accurately on each 

together to form an image picture and therefore detailed image, the images should be enlarged to a certain extent as 

description thereof is omitted. 25 shown in FIG 37 

« 7°? £ F i G ; 3 \ ^ 1 !, With thC dc ^ 0D ™ ih res P cct Specifically,' the star-shape symbol marks 761 and 762, 

to FIG. 5 of the first embodiment in which plural images arc located on the left hand of the billboard for the coffee shop 

joined together to form an image picture, two images are appearing on both images 751 and 752, are suitable as 

specified torn corresponding reduced mages or thumbnails characteristic identical points to be specified. Then, joining 

in the reduced image display area 502. The method to points are specified by moving the cursor, for example to 

specify the images is the same as that shown in the first respective tips 771 and 772 of the star-shape symbol marks 

embodiment in which plural images are joined together to 761 and 762, and clicking the button of the mouse 212 or the 

rorm an image picture. Uke Coordinate data indicative of the joining points speci- 

In FIG. 35 two reduced images or thumbnails 601 and fled are stored in a storage provided inside the joining point 

602 are specified as images of which the distortion is to be specifying part 2403 

frnml'f * 6 redllCed h im . a S eS are discriminated After completion of specifying the identical points, the 

from the other images by inversely (negatively) displaying operator presses the start button 706 to start execution to the 

the file names, or enclosing the reduced images with a bold distortion correction processing. A window, though not 

soec-fi d° ° Perat0r reC ° gniZe ^ rCdUCed imagCS *° Sh0WD here ' fc °P ened ^ a messa B e indicating "Distortion 
p e • correction processing is completed". When an "OK" button 
Alter completion of specifying the reduced images, the in the window is clicked, the distortion correction process- 
operator moves the cursor to an "image correction" menu ing is ended 

^°% t ;; * F "? -iV 1 ^ , the bu !:° n ° f ** Next . description is made to a sequence of processing 

™,in^ ° ^ ? % " ^f*™"****™ 45 steps executed by the image altering part 304 and the display 

mT, Z a T* • P y 1 ° WOf P ^" d0WD meDUS contro1 P art 307 accordin 6 to ,he si * th embodiment. FIG 38 

to select the distortion correcting Junction. Thus, the distor- & a flowchart showing a sequence of processing steps 

TwTpk 3 T " P ° PP ^ ^ djSPlayed " 6XeCUtCd by ^ P«t 304 P and the fcpky 

u m nu. oo. contro , paft ^ Qf (he image processi[lg apparatus 10 q 
Next, the processing contents of the image arranging part 50 according to the sixth embodiment. In the flowchart of FIG 
402 are described. FIG. 36 is an illustration showing another 38, it is first determined whether two or more images are 
example of the display screen on the display pan 308 specified or not (step S901). The operation remains in step 
according to the sixth embodiment. The distortion correcting S901 until two or more images are specified (until step S901 
window 700 includes the distortion correcting work area becomes affirmative), and it is then determined whether 
701, the scale-up button 704, the scale-down button 705, the 5S distortion correction is instructed or not (step S9021 
star, button 706 and the cancel button 707. If distortion correction is not ir^tructed in step S902 (step 
Images specified by the image specifying part 2401 are S902 is negative), it is determined whether other instructions 
displayed in the distortion correcting work area 701. In FIG. such as deletion of images are given or not (step S903) If 
36 images 751 and 752 identical to the reduced images 601 no other instruction is given here (step S903 is negative) the 
and 602 specified in FIG. 35 are displayed at a predeter- 60 procedure returns to step S901, and the subsequent process- 
mined magnification rate. As apparent from FIG. 36, the ing steps are repeated. If any other instruction is given in 
billboard for a coffee shop is distorted so as to have reduced step S903 (step S903 is affirmative), this operation is ended 
magmficanon m a direction to away from the center of the after execution of the corresponding processing (step S904) 

to vf rti c^ortr"d g ;r cs noticeable * reeions close 6S "^rr ion * * step s9 ° 2 (s,ep 

705 perform substantially the same functions, respectively, procedures will be described in detail later 
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performed, specification and arrangement of images, and 
specification of identical points for adjacent images are 
easily done. 

Seventh Embodiment 
Although in the sixth embodiment an identical point in 



It is next determined whether the image arrangement is to 
be altered or not (step S940). If alteration in arrangement is 
required (step S940 is affirmative), the procedure returns to 
step S910 in which the images arc rearranged. If alteration 
is not required in step S940 (step S940 is negative), it is 
determined whether identical points are specified or not 
(step S950). If any set of identical points is specified (step ,. 

S950 is affirmative), coordinate data of the identical points each ls s P ec ^° by moving a cursor to the point and 

specified are saved (step S965). The processing steps S950 ^ clickin S ^ button of the mouse 212 or the like, the 
through S970 are repeated until all the identical points are identical point may be specified using a line, i.e., by speci- 
specified. When all the identical points have been specified 10 f y in 8 identical points on two or more adjacent images in a 
(step S970 is affirmative), it is further determined whether drag-and drop operation of the mouse 212 or the like, as 
specification of any set of identical points is to be altered or discussed in the following seventh embodiment, 
not (step S980). S ; nce me g er , era i structure of the illustrated-document 

If determination is made in step S980 that a specification creating system including an image processing apparatus 
of any set of identical points is to be altered (step S980 is according to the seventh embodiment of the present 
affirmative), the procedure returns to step S950. On the other invention, and the hardware configuration of the image 
hand, if determination is not made in step S980 that speci- processing apparatus 100 are substantially the same as those 
fixation of any set of identical points is to be altered (step of the sixth embodiment, description thereof is omitted here. 
S1980 is negative), it is determined whether the start button Further, since the image processing apparatus 100 includes 
is pressed or not (step S981). If the start button is pressed 20 substantially the same parts as those of the sixth embodi- 
(step S981 is affirmative), the distortion correction is ment except the image altering part 304 and the display 
executed (step S982), and the results of the distortion control part 307, the parts common to those of the sixth 
correction is stored (step S983) to end all the processing. embodiment are also not described here. 

FIG. 39 shows a resulting image for which the distortion Next, the image altering part 304 is described. FIG. 41 is 

correction has been executed. As apparent from FIG. 39, a functional block diagram showing the arrangement of the 
execution of the distortion correction results in correction of image altering part 304 and the surroundings in the image 
the distortion appearing on the lower side of the billboard for processing apparatus 100 according to the seventh embodi- 
the coffee shop. ment of the present invention. As shown in FIG. 41, the 

If the start button is not pressed in step S981 (step 981 is 30 ™age altering part 304 includes an image specifying part 
negative), it is determined whether alteration in image 25 °1> an image arranging part 2502, an identical point 
arrangement is required or 'not (step S990). If alteration in specifying part 2503, a distortion correcting part 2504 and a 
image arrangement is required (step S990 is affirmative), the connection line drawing part 2505. Since parts other than the 
procedure returns to step S910 and the subsequent process- identical point specifying part 2503 and the connection line 
ing steps are repeated. On the other hand, if alteration in 35 drawing part 2505 are substantially the same as those of the 
image arrangement is not required (step S990 is negative), sixth embodiment, description thereof is omitted, 
the procedure returns to step S980 in which determination is The identical point specifying part 2503 has substantially 
made as to whether any set of identical points should be the same structure as that of the identical point specifying 
altered or not. After that, the subsequent processing steps arc part 2403 of the sixth embodiment, in which any one point 
repeated. ^ is specified for each image so that images displayed on the 

Next, the image arrangement processing executed at step display screen of the display part 308 under control of the 
S910 is described in detail. FIG. 40 is a flowchart showing display control part 307, can be joined by referring to the 
processing steps executed by the image arranging pail 2402 P°int, but differs from the identical point specifying part 
of the image processing apparatus 100 according to the sixth 2403 in method of specifying the point. The connection line 
embodiment. If distortion correction is instructed in step 45 drawing part 2505 draws a connection line between the 
S902 of the flowchart of FIG. 38 (step S902 is affirmative), locations specified by the identical point specifying part 
determination is made in the flowchart of FIG. 40 as to 2503. The processing contents of the identical point speci- 
whether an enlargement of the images is possible or not (step tying part 2503 and the connection line drawing part 2505 
S915). If possible (step S915 is affirmative), determination will be described in detail later. 

rtr d C oi a !k t0 ,7 hCtl ? er ^ ? largC c ^^ t 15 instnlcted ° r aot so The image specifying part 2501, the image arranging part 
(step S916). If instructed (step S916 is affirmative), the 2502, the identical point specifying part 2503, the distortion 
enlargement is performed (step S917). On the other hand, if correcting part 2504 and the connection line drawing part 

2505 are embodied, respectively, by the CPU 201 or the like 
executing command processing according to commands 
written in programs such as an OS and an application 
program recorded on recording media such as the ROM 202, 
the RAM 203, the hard disk 205 or the floppy disk 207. 

The display control part 307 controls the display part 308 
to display not only images arranged by the arranging part 



not possible to enlarge the images (step S915 _ 
negative), or if the enlargement is not instructed (step S916 
is negative), the procedure goes to the next step without 
execution of any processing. 

It is next determined whether a reduction of the images is 
possible or not (step S918). If possible (step S918 is 
affirmative), determination is made as to whether the reduc- 

native! the reduction is nerfnrmwl fcion co-mti n„ j . .. . . __ J waDec 



is affirmative), the reduction is performed (step S920). On 
the other hand, if it is not possible to reduce the images (step 

5918 is negative), or if the reduction is not instructed (step 

5919 is negative), the procedure goes to the next step, i.e., 
step 940, without execution of any processing. 

As discussed above, according to the six embodiment of 
the present invention, when correction of image distortion is 



lion line drawing part 2505 as shown in FIG. 42. 

Next, description is made to specification processing of 
identical points and drawing processing of a connection line 
executed by the identical point specifying part 2503 and the 
65 connection line drawing part 2505. FIG. 42 is an illustration 
showing an example of a display screen on the display part 
308 of the image processing apparatus according to the 
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rZtdtsSVch^, p 42 ', ima 8 es 1 1301 1302 « specified is altered, the set of identical points already sped- 
compared to specify a charac ensue point ui common image fled may be released when a set of identical noints is 

the^dentTpo^ . £ft «* W altering pan 304 is described. FIG. 44 is 

provided inside the "identical point spfeS P« 3j e Ten" h^A 0 ^ 8 1116 of ,he 

After completion of the specification processing ofTdeS ™„f « g P „ ^ ^j™ 1 ^ * image 

points, the operator can press the start button toTar P rocess f 8 apparatus 100 according to the eighth embodi- 

execution of distortion correction processing. 20 ment of the P resent invention. As shown in FIG. 44, the 



Thus, the identical points can be specified by the simplest 222? 6 part 304 . includes an specifying part 

way in operation to move the cursor to a point in one image ', ™ una 8 e ""ngiiig part 2602, an identical point 

by operating the mouse or the like, press the button of the s P ec tfywg part 2603, a distortion correcting part 2604, a 

mouse or the like, move the cursor while pressing the button connection line drawing part 2605 and an identical point 

to a corresponding position in the another image, and then 25 s P eciflcation releasing part 2606. 

release the pressed button. The connection line 1321 is Since P arls otner than the identical point specifying part 

preferably drawn by a method such as one described in the 2603 > me connection line drawing part 2605 and the iden- 

second embodiment for joining plural images. tical point specification releasing part 2606 are substantially 

Next, description is made to a sequence of processing me same as mose of the sixth embodiment, and the connec- 

steps executed by the image altering part 304 and the display 30 uon line drawing part 2605 is substantially the same as that 

control part 307 according to the seventh embodiment. FIG. of the seventh embodiment, description thereof is omitted 

executes substantially the same processing steps as those 35 JL*£ ° rt ?<n ™* P ° mt s P ecificati °n 

from S901 to S940 and those fiLs9W to S990 in tnt J? 8 T U ? 6 f 6 " 868 ^cation ° f the previous set 
flowchart of FIG. 38 according to the sixth embodiment S £f™ V f ' 1™ a , Se ' of id ^ ntical ^ « ^ed 
common steps and their description are omitted J, P ™ ageS for WQlch mother set of identical points 

In step S940 of the flowchart of FIG. 38 according to the J* V^f 9 ^ Spe ? fled ^ P r °«ssing contents of the 
sixth embodiment, if alteration in image arrangement is not 40 " Jen,i ? aI j P°^t specification releasing part 2606 will be 
required (step S940 is negative), determination is made in dCSahed m detal1 hter " 

the flowchart of FIG. 43 as to whether the button of the mage specifying part 2601, the image arranging part 

mouse 212 or the like is pressed or not (step S951). If the 2602, ^ lde nUcal point specifying part 2603. the distortion 
button is pressed (step S951 is affirmative), it is determined correcting part 2604, the connection line drawing part 2605 
whether the cursor is positioned on an image or not (step « and the identical point specification releasing part 2606 are 
S952). If the cursor is not positioned on any image (step embodied, respectively, by the CPU 201 or the like execut- 
S952 is negative), an error indication is given (step S953) m g command processing according to commands written in 
and the procedure returns to step S951. On the other hand, programs such as an OS and an application program 
it the cursor is positioned on an image (step S953 is recorded on recording media such as the ROM 202 the 
affirmative), drawing of a connection line is started at the so RAM 203. the hard disk 205 or the floppy disk 207. ' 
P T* I S954) ' Next ' description is made to releasing processing of a set 

After that, the operation remains in step S955 until the of identical points executed by the identical point specifi- 
bulton of the mouse 212 or the like is released. When the cation releasing part 2606. When a set of identical points is 
button is released (step S955 becomes affirmative), the specified for a pair of images, the identical point specifica- 
drawing of the connection line is ended (step S961). After 55 tion releasing part 2606 recognizes whether another set of 
that the procedure goes to step S965 of FIG. 38 according identical points has already been specified for the imaees bv 
to the sixth embodiment referring to whether or not coordinate data of the cone- 

As discussed above, according to the sixth embodiment, sponding identical points are stored in a storage provided 
a connection line is drawn between identical points mside the identical point specifying part 2603. 
specified, so that the operator can not only specify the «> When a new set of identical points is specified the 
ldenUcal points as if he or she drew a line by hand, but also identical point specification releasing part 2606 releases the 
recognize instantaneously whether the identical points are previous identical points by deleting corresponding coordi- 
specined or not. ^ nate data stored, and stores the set of identical points newly 

Eighth Embodiment " ThuS ' the identica l P°inls can be easily changed. 
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Sot IT/ ^l 304 ^ ^ di$Play ron,r01 Pait 5 liDe P art 2705 hav/LstJtfaily th^s^e 

307 according to the eighth embodiment. structure as those of the image specifying parts 2401, 2501 

Since the flowchart of FIG. 45 executes substantially the ^ 26Q\, the image arranging parts 2402, 2502 and 2602 

same processing steps as those fiom S901 to S940 and those me idei "ical point specifying parts 2403,' 2503 and 2603' 

from S965 to S990 in the flowchart of FIG. 38 according to and the . c °nnectioD line drawing part 2405, 2505 and 2605 

the sixth embodiment, the common steps and their descrip- 10 accordi ng to the sixth to eighth embodiments, respectively, 

tion are omitted. Further, the flowchart of FIG. 45 also Such common P arts are not described here, 

executes substantially the same processing steps as those focal lea S& setting part 2707 sets the focal length of 

from S951 to S956 and S965 in the flowchart of FIG. 43, and photographed with a digital still camera 213 or the 

therefore description thereof is omitted as welL ' ' m accorda nce with an operating instruction from the 

THe operation remains in step S955 until the button of the « g^£* P 16 / 0 ^ ^ «' by 

mouse 212 or the like is released. If the button is released o^^S*^^^^, . ^ ^ °' 

(step S955 becomes affirmative, it is then determined foSk^h, 1;^ , j° <*mera types and their 

whether identical points are specified for the plural imaees Zt\ g ^ ' Pre-stored, by inputtmg mformation 

or not (*te* SQStt if ,h» ;T . i . Plural raiages regarding the camera type, whether a zooming lens is used 

or not(step ;>y56). Jt the identical points are not specified for »nH th* Hir» n,. ~L • . , ■ , . 

snins to step b!*61 without execution of any processing. n,» ; m ™ _ -c • , . 

» j • ■ . ,. me CFU 201 or the lie executing command processing 

As discussed above, according to the eighth embodiment, according to commands written in programs such as an OS 

whf°h a ^ "Jc"^ Points is specified for images for and an application program recorded on recording media 

which another set of identical points has already been Such as the ROM 202, the RAM 203, the hard disk 205 or 

specified, the identical point specification releasing part the floppy disk 207 

i 6 e 0 c 6 ified 6 T,n5^ wH ^ % ° f ^T- descri P tion » ™* e to focal-length setting process- 

spec fied so that when specifying a new set of identical ing executed by the focal length setting part 2707 FIG 47 

points, the operator can change the specification of identical is an illustration showing an example of Stay Screen on 

SS^XSEP S r fi T rati ° n ^ heD * ^^P"* 3 °« of the image U^S^TlZ 

specifying a set of identical points for the first time without according to the ninth embodiment. In FIG 47 a distortion 

SenS oi^ S P eClficatl0n 0f "» P»*« ■* of correcting wmdow 1800 includes a distortion colcting 

identical pomts. work are a 1801 a zQom ^ lm a ^ * 

Ninth Embodiment switching part 1803, a scale-up button 1804, a scale-down 
40 button 1805, a start button 1806 and a cancel button 1807 

Although the above sixth to eighth embodiments for The distortion correcting work area 1801 the scale-uo 

correcting image dlS tortion have been made for images button 1804, the scale-down button 1805 ^ tt start button 

CtaleS, iT ,h h StJ1 ^ mera 213 ^ 3 fiXEd 1806 ^ CaDCel bm °» 1807 havc -bsUtiaHy ie same 

S en E ,h' ' ^° ve 2 mb °^ mentS ™ e ^ ^ struchlre « ^ °f the distortion correcting work area 701 
u^aldiftlrnnT^^ ^ ™ a 8 es f^^rrecting 45 the scale-up button 704, the scale-down butfon 705, the star 

miage distortion are those photographed with a digital still button 706 and the cancel button 707 according to the sixYh 

camera 213 having variable focal lengths, the focal length embodiment, respectively. Such common plZ a e not 

may be set for proper distortion correction of images, as described here. P 

S^he tn^7™l ninth f TH b ^ ent - ^ ^ Z00m SWltChiDg P"' 1802 zoom 

Since ihe general structure of the illustrated-document 50 switching, alternatives related to whether an image to be 

creating system including an image processing apparatus joined with another has been photographed in the zoom 

anT tne 3 hi mD,h mention mode or not. When no zoom is uLi fonhe imaged mod" 

3 nl =L ,n« c °f S"™ 110 " of the image processing is switched to "standard" S1813. On the other hand, when 

E r fflb rf yth f eSameaSlh0Se ° ftheSixlh the Z00m * USed for ,he ima S e - mode is switched o 

embodiment, description thereof b omitted here. Further, 55 "maximum" 1814. The zoom mode is thus switched 

X l e sl a e 8 oaT a e T g aP fT niS 1°° T 1 ?* SUbStan " A " hon 8 h * ^ ^bodiment switching is enabled by 

a functional block diagram showmg the arrangement of the of digital still cameras, these information related to S 

pSSIT^ ^surroundings in the image of thecal length be ^X^t^S 

pressing apparatus 100 according to the nmth embodiment In this case, the focal length may be automatically se t b V 

rLrinJ^nT^T- Asshown m mG - 46 ' ^ ™ a ge reading the information related to setting, of the foci ^length 

altering part 304 includes an image specifying part 2701, an 65 which is added to respective image data 

2^n1\ ar HT^ Part27te ' aflident ^ p0intSpeci ^ ng P art ™ e ^Ktiing part 1803 displays a list of the 

2703. a chstortton correeung part 2704, a connection line names of seiectable digftafstill c^eZ?£ orator ^ 
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select, out of the listed names, the name of a digital still 
camera with which an image to be joined with other images 
has been photographed. The camera switching is thus per- 
formed. 

Next, description is made to a sequence of processing s 
steps executed by the image altering part 304 and the display 
control part 307 according to the ninth embodiment FIG. 48 
is a flowchart showing a part of processing steps executed by 
the image altering part 304 and the display control part 307 
of the image processing apparatus 100 according to the ninth i° 
embodiment. 

Since the flowchart of FIG. 48 executes substantially the 
same processing steps as those from S901 to S910 and those 
from S940 to S990 io the flowchart of FIG. 38 according to 
the sixth embodiment, the marks of the common steps and 15 
their description are omitted. The flowchart of FIG. 48 also 
executes substantially the same processing, steps as those 
from S911 to S920 in the flowchart of FIG. 40 according to 
the sixth embodiment and therefore description thereof is 
omitted as well. 20 

In the flowchart of FIG. 48, if step S919 is negative or 
after completion of step S920, determination is made as to 
whether focal-length setting processing is instructed or not 
(step S931). If instructed (step S938 is affirmative), the 
focal-length setting processing is performed (step S932). On 25 
the other hand, if not instructed (step S931 is negative), the 
procedure goes to the next step, i.e., step S940, without 
execution of any processing. 

As discussed above, according to the ninth embodiment, 30 
the focal length setting part 2707 sets the focal length with 
which an image has been input, while the distortion correct- 
ing part 2704 corrects distortion of the image based on the 
focal length set for use in distortion correction processing of 
the image. This makes it possible to edit images without 
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joining means for joining the at least two specified images 
by referring to the joining points specified by said 
joining point specifying means; 
wherein said display control means controls said display 
device to display both the images arranged by said 
image arranging means and the connection line drawn 
by said connection line drawing means. 

2. An image processing apparatus, comprising: 
a display device configured to display images on a display 

screen such that the images can be edited and pro- 
cessed; 

image specifying means for specifying at least two images 

to be joined together; 
image arranging means for arranging the images specified 

by said image specifying means in a desired order; 
display control means for controlling said display device 
to display the images arranged by said image arranging 
means; 

joining point specifying means for specifying respective 
joining points for at least two of said specified images; 
joining means for joining the at least two specified images 
by referring to the joining points specified by said 
joining point specifying means; and 
joining-point specification releasing means for releasing ' 
specification of a previous specified set of joining 
points for a pair of images when a new set of joining 
points is specified for the pair of images. 

3. The image processing apparatus as set forth in claim 1, 
further comprising: 

focal length setting means for setting a focal length of an 

image to be specified; and 
image correcting means for correcting a specified image 
based on the focal length set by said focal length setting 
n if the images have been input means, 
by an input device (an image pick-up device) with different 4 - T/* e image processing apparatus as set forth in claim 1, 

focal lengths. wherein said display control means controls the display 

Numerous additional modifications and variations of the device 10 the images vertically, 

present invention are possible in light of the above teach- 40 5 " ^ imag6 P roccssin g apparatus as set forth in claim 1, 
ings. It is therefore to be understood that within the scope of wnerein said display control means controls the display 
the appended claims, the present invention may be practiced device ta ^P 1 ^ the images horizontally. 

" - * A "" ,U " A 1 image processing 



otherwise than as specifically described herein. 

This document claims priority and contains subject matter 
related to Japanese patent applications No. 10-14413, No. 45 
10-196278, and No. 10-196280 filed in the Japanese Patent 
Office on Apr. 10, 1998, Jul. 10, 1998 and Jul. 10, 1998, 
respectively, and the entire contents of which are hereby 
incorporated by reference. 

What is claimed is: 50 
1. An image processing apparatus, comprising: 
a display device configured to display images on a display 
screen such that the images can be edited and pro- 
image specifying means for specifying at least two images S5 

to be joined together; 
image arranging means for arranging the images specified 

by said image specifying means in a desired order; 
display coQtrol means for controlling said display device 60 
to display the images arranged by said image arranging 

joining point specifying means for specifying respective 
joining points for at least two of said specified images; 

connection line drawing means for drawing a connection 65 
line between joining points specified by said joining 
point specifying means, and 



6. A method for controlling ; 
apparatus, comprising steps of: 

; displaying images on a display screen; 

specifying at least two images to be joined together; 
arranging the images specified in said specifying step in 

a desired order; 
displaying the specified images arranged in said arranging 
step; 

specifying respective joining points for at least two of the 

specified images; 
drawing a connection line between the joining points 

specified in said joining point specifying step; and 
joining the at least two images by referring to the joining 

points specified in said joining points specifying step; 
wherein said arranged images displaying step displays 

both the images arranged in said arranging step and the 

connection line drawn in said connection line drawing 

7. A method for controlling an image processing 
apparatus, comprising steps of: 

displaying images on a display screen; 
specifying at least two images to be joined together; 
arranging the images specified in said specifying step in 
a desired order; 
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displaying the specified images arranged in said arranging 
step; 

specifying respective joining points for at least two of the 

specified images; 
joining the at least two images by referring to the joining s 

points specified in said joining points specifying step; 

releasing specification of a previous specified set of 
joining points for a pair of images when the specifying 
step is performed for the pair of images. 

8. The method as set forth in claim 6, further comprising 
steps of: 

setting a focal length of an image to be specified; and 
correcting the specified image. based on the focal length 15 
set in said focal length setting step. 

9. The method as set forth in claim 6, wherein said step 
of displaying the arranged images comprises displaying the 
images vertically. 

10. The method as set forth in claim 6, wherein said step 20 
of displaying the arranged images comprises displaying the 
images horizontally. 

11. A computer-readable recording medium storing com- 
puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 25 

displaying images to be specified on a display screen; 
arranging specified images in a desired order; 
displaying the images arranged in said arranging step; 
joining at least two of the images by referring to respec- 30 

tive user specified joining points of the images; and 
drawing a connection line between the joining points; 
wherein said arranged image displaying step displays of 

the images arranged in said arranging steps and the 
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12. A computer-readable recording medium storing com- 
puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 

displaying images to be specified on a display screen; 

arranging specified images in a desired order; 

displaying the images arranged in said arranging step; 

joining at least two of the images by referring to respec- 
tive user specified joining points of the images; and 

releasing specification of a previous specified set of ' 
joining points for a pair of images when the specifying 
step is performed for the pair of images. 

13. The computer readable recording medium as set forth 
in claim 11, the computer instructions further controlling an 
image processing apparatus to perform the step of: 5 

correcting a specified image based on a focal length of the 
specified image. 

14. The computer readable recording medium as set forth 
in claim 11, wherein said step of displaying the arranged 
images comprises displaying the images horizontally. 5 

15. The computer readable recording medium as set forth 
in claim 11, wherein said step of displaying the arranged 
images comprises displaying the images vertically. 

16. An image processing apparatus, comprising: 

a display device configured to display images on a display 6 

screen such that the images can be edited and pro- 
image specifying means for specifying at least two images 

photographed at different photographing positions; 6 
image arranging means for arranging the images specified 
by said image specifying means; 



display control means for controlling said display device 
to display the specified images arranged by said image 
arranging means; 
point specifying means for specifying respective points 
for at least two of the specified images displayed by 
said display control means; 
connection line drawing means for drawing a connection 
line between the points specified by said point speci- 
fying means; 

distortion correcting means for correcting distortion of the 
at least two images by referring to the points specified 
by said point specifying means; and 
wherein said display control means controls said display 
device to display both the images arranged by said 
image arranging means and the connection line drawn 
by said connection line drawing means. 

17. An image processing apparatus, comprising: 
a display device configured to display images on a display 

screen such that the images can be edited and pro- 
image specifying means for specifying at least two images 

photographed at different photographing positions; 
image arranging means for arranging the images specified 

by said image specifying means; 
display control means for controlling said display device 
to display the specified images arranged by said image 
arranging means; 
point specifying means for specifying respective points 
for at least two of the specified images displayed by 
said display control means; 
distortion correcting means for correcting distortion of the 
at least two images by referring to the points specified 
by said point specifying means; and 
point specification releasing means for releasing specifi- 
cation of a previous specified set of points for a pair of 
images when a new set of points is specified for the pair 
of images. 

18. The image processing apparatus as set forth in claim 
16, further comprising: 

focal length setting means for setting the focal length of 
the images to be specified; 

wherein said distortion correcting means corrects distor- 
tion of the specified images based on the focal length 
set by said focal length setting means. 

19. A method for controlling an image processing 
apparatus, comprising: 

displaying images on a display screen; 

specifying at least two images photographed at different 

photographing positions; 
arranging the images specified in said image specifying 

step; 

controlling display of the images arranged in said image 
arranging step; 

specifying respective points for at least two of the images 
displayed in said display control step: 

drawing a connection line between the points specified in 
said point specifying step; and 

correcting distortion of the at least two images by refer- 
ring to the points specified in said point specifying step; 

wherein said control step controls display of both the 
images arranged in said image arranging step and the 
connection line drawn in said connection line drawing 
step. 
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unrolling an image processing 



20. A method for ( 
apparatus, comprising: 

displaying images on a display screen; 

specifying at least two images photographed at different 
photographing positions; arranging the images speci- 
fied in said image specifying step; 

controlling display of the images arranged in said image 
arranging step; 

specifying respective points for at least two of the images 
displayed in said display control step; 1 

correcting distortion of the at least two images by refer- 
ring to the points specified in said point specifying step; 

releasing specification of a previous specified set of points i 
for a pair of images when a given set of points is 
specified for the pair of images in said point specifying 

21. The method as set forth in claim 19, further compris- 
ing a step of: 
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26. A computer-readable recording medium storing com- 
puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 

specifying respective joining points for at least two 

images displayed on a display device; 
joining the at least two images by referring to the joining 
points; and 

releasing specification of a previous specified set of 
joining, points for a pair of images when a new set of 
joining points is specified for the pair of images. 

27. A computer-readable recording medium storing com- 
puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 

specifying respective points for at least two images dis- 
played on a display device; 
correcting distortion of the at least two images by refer- 
ring to the points specified by said point specifying 
mj means; and 

setting a focal length of an image to be specified, drawing, on the display device, a connection line between 

wherein said distortion correcting step corrects distortion the points specified by said point specifying means. 

of me specified ^mage based on the focal length set in 28. A computer-readable recording medium storing com- 

22^ ™,,l?^, 8 5' „• Puter mstnicdons for controlling an image processin|appa- 

r;.^." m .^7 eadab i e r eco rdmg medium storing com- ™ ratus to perform the steps of: 

specifying respective points for at least r\ 



puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 

displaying images to be specified on a display screen; 
arranging specified images in a desired order; 
displaying the images arranged in said arranging step; 3 ° 
correcting distortions of least two of the images by 
referring to respective user specified points of the 
images; and 

drawing a connection line between the specified points, 35 
wherein said display control step controls display of both 
the images arranged in said image arranging step and 
the connection line drawn in said connection line 
drawing step. 

23. A computer-readable recording medium storing com- to 
puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 

displaying images to be specified on a display screen; 
arranging specified images in a desired order; 
displaying the images arranged in said arranging step; 45 
correcting distortions of least two of the images by 

referring to respective user specified points of the 

images; and 

releasing specification of a previous selected set of points 
for a pair of images when a new set of points is 50 
specified for the pair of images. 

24. The computer readable recording medium as set forth 
in claim 22, the computer instructions further controlling an 
image processing apparatus to perform the step of: 

correcting distortion of a specified image based on the 
focal length of the image. 

25. A computer-readable recording medium storing com- 
puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 

specifying respective joining points for at least two 

images displayed on a display device; 
joining the at least two images by referring to the joining 

points; and 

connection line drawing means for drawing on the display 
device, a connection line between joining points speci- 
fied by said joining point specifying means. 



images dis- 
played on a display device; 
correcting distortion of the at least two images by refer- 
ring to the points specified by said point specifying 
means; and 

point specification releasing means for releasing specifi- 
cation of a previous specified set of points for a pair of 
images when a new set of points is specified for the pair 
of images. 

29. A computer-readable recording medium storing com- 
puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 

displaying images to be specified on a display screen; 

displaying the specified images in both a horizontal and a 
vertical arrangement on the display screen; 

displaying the specified images only in a user selected one 
of the horizontal and vertical arrangements of the 
specified images; 

drawing, on the display screen, a connection line between 
user specified joining points; and 

joining least two of the images displayed in said step of 
displaying the specified images only in a user selected 
arrangement, by referring to respective said user speci- 
fied joining points of the images. 

30. A computer-readable recording medium storing com- 
puter instructions for controlling an image processing appa- 
ratus to perform the steps of: 

displaying images to be specified on a display screen; 
displaying the specified images in both a horizontal and a 

vertical arrangement on the display screen; 
displaying the specified images only in a user selected one 

of the horizontal and vertical arrangements of the 

specified images; 
drawing, on the display screen, a connection line between 

user specified points; and 
correcting distortion of least two of the images displayed 

in said step of displaying the specified images only in 

a user selected arrangement, by referring to respective 

said user specified points of the images. 
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:t forth in claim 



31. The image processing apparatus a; 
1, further comprising: 

means for standardizing the brightness of the specified 
images by reference to the brightest one of the specified 
images. 

32. The image processing apparatus as set forth in claim 
1, further comprising: 

means for standardizing the brightness of the specified 
images by reference to the first specified one of the 
specified images. ] 

33. The method as set forth in claim 6, furthei 
the step of: 

standardizing the brightness of the specified images by 
reference to the brightest one of the specified images. 

34. The method as set forth in claim 6, further comprising 
the step of: 

standardizing the brightness of the specified images by 
reference to the first specified one of the specified 
images. 

35. The computer readable recording medium as set forth 
in claim 11, the computer instructions further controlling the 
image processing apparatus to perform the step of: 

standardizing the brightness of the specified images by 
reference to the brightest one of the specified images. 25 u^^' 

36. The computer readable recording medium as set forth 001 
in claim 11, the computer instructions further controlling the 
image processing apparatus to perform the step of: 

standardizing the brightness of the specified images by 
reference to the first specified one of the specified 
images. 

37. The image processing apparatus as 
16, further 
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44. The image processing apparatus as set forth in claim 
43, wherein said characteristic is color. 

45. The image processing apparatus as set forth in claim 
43, wherein said characteristic is line thickness. 

46. The image processing apparatus as set forth in claim 
43, wherein said characteristic is operator selectable. 

47. The image processing apparatus as set forth in claim 
43, wherein said characteristic is automatically selected by 
taking into account a color used in the image. 

48. The method as set forth in claim 14, wherein said 
connection line has a characteristic such that it can be readily 
discriminated from the remainder of the image. 

49. The method as set forth in claim 48, wherein said 
characteristic is color. 

50. The method as set forth in claim 48, wherein said 
characteristic is line thickness. 

51. The method as set forth in claim 48, wherein said 
characteristic is operator selectable. 

52. The method as set forth in claim 48, wherein said 
characteristic is automatically selected by taking into 
account a color used in the image. 

53. The computer readable recording medium as set forth 
in claim 22, wherein said connection line has a characteristic 
such that it can be readily discriminated from the remainder 

The computer readable recording medium as set forth 
in claim 53, wherein said characteristic is color. 

55. The computer readable recording medium as set forth 
in claim 53, wherein said characteristic is line thickness. 

56. The computer readable recording medium as set forth 
in claim 53, wherein said characteristic is operator select- 



it forth in claim 



:d image 



57. The computer readable recording medium as set forth 
in claim 53, wherein said characteristic is automatically 
selected by taking into account a color used in the image. 

58. The image processing apparatus as set forth in claim 
1, wherein said image arranging means comprises means for 
displaying options for arranging the specified images in one 
or more arrangements, and means for displaying the images 
arranged according to a selected one of said options. 

59. The method as set forth in claim 6, wherein said in 
arranging step comprises displaying options for a 
the specified images in one or more arrangements, and then 
displaying the images arranged according to a selected one 

standardizing the brightness of the specified images by 45 °^ sa '^ options, 
reference to the brightest one of the specified images. 60. The computer readable recording medium as set forth 

40. The method as set forth in claim 19, further compris- m claim U > therein said image arranging step comprises 
ing the step of: displaying options for arranging the specified images in one 

standardizing the brightness of the specified images by ° r m ° re ^^Sements, and then displaying the images 
reference to the first specified one of the specified so arraa 8 ed according to a selected one of said options, 
images. 61. The image processing apparatus as set forth in claim 

41. The computer readable recordings medium as set forth } 6 ' T herei ° ^ ima S e arranging means comprises means 
in claim 22, the computer instructions further controlling the displaying options for arranging the specified images 



means for standardizing the brightness of the specified 
images by reference to the brightest one of the specified : 
images. 

38. The image processing apparatus as set forth in claim 
16, further comprising: 

means for standardizing the brightness of the specified 
images by reference to the first specified one of the " 
specified images. 

39. The method as set forth in claim 19, further compris- 
ing the step of: 



image processing apparatus to perform the step of: 
standardizing the brightness of the specified images by : 
reference to the brightest one of the specified images. 

42. The computer readable recording medium as set forth 
in claim 22, the computer instructions further controlling the 
image processing apparatus to perform the step of: 

standardizing the brightness of the specified images by ( 
reference to the first specified one of the specified 

43. The image processing apparatus as set forth in claim 
1, wherein said connection fine has a characteristic such that 



arrangements, and means for displaying the 
images arranged according to a selected one of said options. 

62. The method as set forth in claim 19, wherein said 
image arranging step comprises displaying options for 
arranging the specified images in one or more arrangements, 
and then displaying the images arranged according to a 
selected one of said options. 

63. The computer readable recording medium as set forth 
in claim 22, wherein said image arranging step comprises 
displaying options for arranging the specified images in one 
„r ™ arrangements, and then displaying the images 
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ABSTRACT 



A system and method for capturing image data captures 
images responsive to programmed sequences. The 
sequences are performed on a periodic basis as well as in 
response to inputs corresponding to alarm conditions and 
transactions conducted at automated banking machines or 
other devices. Image data may also be captured in response 
to image conditions including the sensing of motion or the 
loss of usable video from selected cameras. Image data is 
stored in connection with data corresponding to circum- 
stances associated with each triggering event. Stored image 
data may be searched by one or more parameter. Param- 



eters include data stored ii 



association with each image, 



types of events causing image data to be stored, as well a: 
other image conditions in stored images. 



70 Claims, 79 Drawing Sheets 
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Apply all recast configuration changes made to the system. 
Hole that during application of the changes, recording el 
Images will be temporarily hated. 


V 




V 


Help 


Access to on-llaa help. 






V 
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Buttons 


Description 


n<] 

HD 


Use these button to go to the beginning or the end of a 
series of events or images (depending on the direction 
of the arrow). 


L22J 


Use these buttons to go forward or backward by ten events 
or images (depending on the direction of the arrow). 


[El. 




H 


Use these buttons to go forward or backward by one event 
or image (depending on the direction of the arrow). 


EE 




Jump to Image # 
Jump to Event # 


Type the number of the image or events you want to view, 
and click on this button to display that image as the large 
image. The image or event number displays next to each 
thumbnail frame (for example, Image 7 of 48). 


Save Comments 


To store comments with the image, type your comments in 
the comments field, and click on this button. The next time 
this image is retrieved, the comments display with the 
image. 


J 
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TON No. 43928 
Time/Date: 09:28:42 

07:13:1998 
Acct. No. 62817041867 



Event 528 



Image 1 of 5 



EI [>D 



Jump to Image # 
Jump to Event # 
Save Comments 
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Motion #1 
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Input #2 


Motion #2 


Camera Block #2 


Transaction #2 


Input #3 


Motion #3 


Camera Block #3 


Transaction #3 


Input #4 


Motion #4 


Camera Block #4 


Transaction #4 


Input #5 








Input #6 




| Daily Program! 




Input #7 






Input #8 
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User Access Assignments 
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Edit Individual E-mail address 



|Chrls DiVita - dMtc@diebold.com | | 


| | E-mail Address 


|Edlt | Individual E-moil Address 


| | Name 


| Delete | irtdMdual E-mail Address 


| Update | E-mail Address 


Edit E-mail Groups 


| Buckeyes | | 


| Edit | Members of Group \Md\ E-mail Group 


[Delete | E-mail Group 
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<&EB€ML& Digital Video Recorder 

or click on an Icon from the toolbar! 
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AccuTrack Toolbar Items 



ITEM | NAME | COMMENTS 


MENU BOX (11 


| Make Selection — »M 


Menu box with drco-down 
arrow 


Click on the drop-down arrow lor a Bsl ol main menus items. 
The access level ol the user determines the displayed 
meous items. 


MAIN MENU ICONS (1] 








Home Icon 


Click on the Homo icon to go to the AccuTrack home page. 






7^376 


Logon icon 


Click on the Logoff icon to kg off the AccuTrack. The 
AccuTrack logon page re-displays. 






r 378 


Perform Image Search lean 


Click on toe Perform Image Search Icon (binoculars) 
to access the Image Search function. 






^380 


Camera Check Icon 


Click on die Camera Check icon lo access the Camera 
Check function. 




^382 


System Configuration Icon 


Click on the System Configuration icon to display (he DVR 
System Conflg submenu In a menu panel on the letl side 
ol the page. 


[^384 


DVR System Tools Icon 


dick on the Tools icon lo display the DVR Tools submenu 
In e menu panel on the left side of the page. 






Help icon 


CLick on the Help menu to display on-line help. 
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AccuTrack Toolbar Items (continued) 



ITEM | NAME | COMMENTS 

STATUS 121 


Status^y 388 


Display status warning 


Toggle (dick once to torn oil and dick agale to turn 
oa) on do word Status to display tutu warning 
messages ia a pop-up dialog box 




Green tigM status icon 


AccuTrack It capturing images properly; 


1 ' ^ 1 1 


Thermometer status Icoe 


AccuTrack It Rearing maximum storage capacity and 
not storing images. The disk it ha 


JJ-394 


Yallow hand (caution) 
statu Icon 


AwuTrack » running wllk errors. Check Me diagnostic! 


HT 396 


Red distorts stilus Icon 


Pendlag changes have out been epplied. 


|^^-398 


Stop sign stilus Icon 1 


An application error has occurred. Check the diagnostics 

NOTE 

• This Icon may display temporarily 
B yon attempt to use AccuTrack 
while AccuTrack It applying 
changes. Wait a lew moments 
and Dim try again. 

e U you are logged oa from a 
remote PC. tHs Icon may 
Indicate i communication! grab lent. 
AccuTrack may still be 
tancQonlag correctly. 


|1 1 Each main menu 1cm It represented by toil hi lbs drop-down msiu. Cfck on i main msau Icon or 
select Irani tie drop-down mtaa to iccess the mens option. 

[2] The currant tutus leoi displays eich time you nrtnuh your Web page or navigate to a new AccuTrack page. Sometimes 
AecuTraek updates tte status while yon remain oa the uroe AccuTrack page. 
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Image Type Override Priority Retain for 

Normal Q | Delete First H 1 7 Days H 

Alarm □ | Delete Last M 1 14 Days |V| 

Transaction □ j Delete If Necessarfrrj 1 90 Pays j^j 



402 Example of Auto Delete Settings for Image Types 
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Setup Auto Delete 








{Auto Delete} is Enabled when this box Is checked 


0 








When you would like{Auto Delete} to begin deleting images? 








*Begln deleting images when the DVR disk space Is below 

1 MB is 1 .048.576 bvtes or 1 .024fKB> Kilobytes. Ro5 I MB 








When would you like{Auto Delete} to stop deleting images? 








*Stop deletlnq Images when the DVR disk space Is above 

11000 1MB 








Image Type Override Priority 


Retain for 








Normal □ 


1 delete Last H 


1 7 Days M 








Alarm □ 


| delete Last \w\ 


1 14 Days |^r| 








Transaction □ 


| delete Lost |^| 


1 90 Days 








| Update || Reset | 
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Default Settings 
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Enable Security 
0 


Check this box to enable Image Security. If the 
box is not checked, all other parameters below 
are Ignored and no Security Signatures will be 
applied to images. 


Normal Images 
□ 


Causes DVR to perform security algorhythms 
on Normal Images. 


Alarms 
0 


Causes DVR to perform security algorhythms 
on Alarm Images such as Input and Motion 
Alarms. 


Transactions 
SI 


Causes DVR to perform security algorhythms 
on Images captured from ATM Transactions. 


| Update || Reset | 
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Camera Setup 



Cam Description 


Cam Description 


U1 1 VESTIBULE DOOR 1 


1 3 I Unavailable I 


02 1 1 


14 I Unavailable I 


03 1 1 


15 I Unavailable I 


04 1 1 


1 6 I Unavailable I 


Ob 1 1 


1 7 I Unavailable | 


Ob 1 1 


1tS | Unavailable I 


0/ 1 1 


1 9 I Unavailable I 


08 1 1 


20 I Unavailable I 


09 1 1 


Z~i (Unavailable | 


10 I I 


22 (Unavailable | 


11 I I 


23 (Unavailable | 


12 I I 


24 (Unavailable I 


Click on a Camera Number 


to perform an image check! 



Update Camera Names { Cancel Form Changes" 



G217 
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Output Setup 



OP Description 


OP Description 


U1 | VESTIBULE LIGHTS | 


05 | UNAVAILABLE | 


UZ I0utput2 | 






07 1 UNAVAILABLE | 


04 |Output4 | 


08 | UNAVAILABLE | 






| Update Output Names | 


Cancel Form Changes | 



G218 



FIG. 37 



Input Setup 



IP Description 


IP Description 


U1 | TELLER PANIC BUTTON | 


05 1 UNAVAILABLE | 




06 | UNAVAILABLE ~] 


03 |lnput #3 | 


07 [UNAVAILABLE | 


IN |lnput#4 | 


08 | UNAVAILABLE 1 






| Update Input Names || 


Cancel Form Changes | 
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Input Setup Page 
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ATM Monitoring Setup 



ATM #1 


Enabled □ 


Date Rate: 1 4800 |-J 






Encodlng:| ASCII M 


ATM Name: |ATM1 




| Stops Bits:[T~F1 


Port Number: |2 M 




Paritv:l Even M 


Protocol: |ExpressBus 


hi 


Data Bits: 1 7 M 


Messagel ExpressBus 
Format:' 


hi 


CRC Preset: ro~R 


ATM Address: | | 




NRZ:|NRZ M 




Port Timeout: 1 30 | 



| Update H Reset || Back | 



LATM Monitoring Setup Page 
414 
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Edit E-mail address 



| Cole, Ted - colet@dlebold.com ~R 



| Edit | Individual E-mail Address 



U Name 



Delete Individual E-mail Address 



I Update 1 E 



Edit E-mail Groups 



□3 


1 1 


| Edit | Members of Group 


| Add | E-mail Group 


| Delete | E-mail Group 





T E-mail Setup Page (for Individuals) 

™^ FIG. 40 
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Sequence Programing 



Daily Program Setup 



input Setup 




Sequence Progamming Page 
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Back to Sequence Setup 
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Daily Program Page 
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Everyday #1 Page 
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Input Setup 



# 


DESCRIPTION 


STATE 


ENABLE 


DISABLE 


SEQUENCE 


1 


Input #1 


ENABLED 


1 Enable 1 


1 Disable I 


| Configure #1 | 


2 


Input #2 


ENABLED 


1 Enable 1 


[ Disable | 




3 


Input #3 


ENABLED 


1 Enable 1 


| Disable | 




4 


Input #4 


ENABLED 


1 Enable 1 


1 Disable I 


| Configure #4 • | 


5 


Not Configured 


DISABLED 


Not 
Implemented 


Not 
Implemented 


| Configure #5 | 


6 


Not Configured 


DISABLED 


Not 
Implemented 


Not 
Implemented 


| Configure #6 | 


7 


Not Configured 


DISABLED 


Not 
Implemented 


Not 
Implemented 


| Configure #7 | 


8 


Not Configured 


DISABLED 


Not 
Implemented 


Not 
Implemented 


| Configure #8 | 



m\ i 
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Input Setup Block (Sequence Programing page) 

FIG. 46 
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% OVR Administrator - Netscape 



if) {gig M * W *© Status: 8 




EE 



y Schedule for Input #2 Page 
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Input #2 Page 
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Motion Setup 



# 


CAMERA 


SIATE 


ENABLE 


DISABLE 


SETUP 


SEQUENCE 


1 


06- 


Enabled 


Enable #1 1 




1 Setup #1| 


I Configure #1 I 


2 


12- 


Enabled 


Enable #2| 


blsable #2l 


f SetUD #2 1 


| Configure #2 | 


3 


03-camaa 
3 


Enabled 


Enable #31 


iDisable # j 


| Setup #3 1 


I Configure #3 I 


4 


01 -My 
Rrst 
Cam&a 


Enabled 


Enable #4 1 


iDisable #4 


I Setup #4 1 





Camera Block Setup 

| 1 CAMERA || STATE || PI ABLE || DISABLE || SETUP ||SEQUEHCe1 
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Motion Setup ■ 1 




Motion Setup > I Page 
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Motion 


Camera 


Compression 


PreAlarm Images 
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Detection Motion 01 
1 OB-BackDoor M 


O Unrest quality, smallest Gte size 
0 Good quality, medium file the 
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Number of Images to 
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[2 R 
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■ Use AVI 
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Motion #1 Page 
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AccuGuard* Blocked Camera Setup 






#| CAMERA 


| STATE | ENABLE | DISABLE | SETUP | SEQUENCE 








Select From Available Cameras | 
["sejecTI 


31- Camera #1 


H 










Transaction Setup 










# 


ATM 


STATE 


SETUP 


1 SEQUENCE 










1 


A1M1 


Enabled 


Se1up#l 


P Configure #1 










2 


ATM 2 


Enabled 


Setup #2 


ll Configure #2 | 








3 


ATM 3 


Enabled 


Setup #3 


P Configure #3 | 








4 


ATM 4 


Enabled 


Setup #4 


( Configure #4 1 










|| Document Done 
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Before Selecting the Camera 
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AccuGuard* Blocked Camera Setup 





CAMERA 


STATE 


ENABLE 


DISABLE 


SETUP 


SEQUENCE II 


r 


Camera #1 


Enabled 


Enable 


Disable 


Setup 


Configure || 



Transaction Setup 


1 # 


ATM 


STATE 


SETUP 


1 SEQUENCE J 




ATM 1 


Enabled 


Setup #1 


\ Configure #1 || 


2 


ATM 2 


Enabled 


Setup #2 


p Configure #2 j] 


3 


ATM 3 


Enabled 


Setup #3 


\ Configure #3 || 


4 


ATM 4 


Enabled 


Setup #4 


| Configure #4 || 
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After Selecting the Camera 



FIG. 54 
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5~=»F1 f ^AT«»Q Status: fg 



2 


Hock_2_Description 


8lock_2_Srate 


Not 
Implemented 


Not 
Implemented 


Not 
Implemented 


Not 
Implemented 


3 


aock_3_Description 


Block_3_State 


Not 
Implemented 


Not 
Implemented 


Not 
Implemented 


Not 
Implemented 


4 


BlockjtDescrlptlon 


Block_4_State 


Not 
Implemented 


Not 
Implemented 


Not 
Implemented 


Not 
Implemented 







Transaction Setup 




# 


ATM 


STATE 


SETUP 


[SEQUENCE 


1 


Not Configured 


Not Configured 


Setup #1 




2 


Not Configured 


Not Configured 


Setup #2 


|| Configure #2 | 


3 


Not Configured 


Not Configured 


Setup #3 


Configure #3 | 


4 


Not Configured 


Not Configured 


Setup #4 


\ Configure #4 | 



H Document Done" 
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Transaction Setup Block (Sequence Programming page) 
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Transaction Page #1 
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images Page Including AVI Symbols for AVI Files 
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Standard Events Page 

FIG. 64 



10/15/2003, EAST Version: 1.04.0000 



U.S. Patent Jun. 24, 2003 Sheet 61 of 79 US 6,583,813 Bl 




10/15/2003, EAST Version: 1.04.0000 



U.S. Patent 



Jun. 24, 2003 



Sheet 62 of 79 



US 6,583,813 Bl 




10/15/2003, EAST Version: 1.04.0000 



U.S. Patent 



Jun. 24, 2003 



Sheet 63 of 79 



US 6,583,813 Bl 




10/15/2003, EAST Version: 1.04.0000 



U.S. Patent Jun. 24, 2003 Sheet 64 of 79 US 6,583,813 Bl 




FIG. 68 



10/15/2003, EAST Version: 1.04.0000 



U.S. Patent Jun. 24, 2003 Sheet 65 of 79 US 6,583,813 Bl 




FIG. 69 



10/15/2003, EAST Version: 1.04.0000 



U.S. Patent Jun.24,2003 Sheet 66 of 79 US 6,583,813 Bl 
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Search Results Pages and Views 



Page 


Figure/View 


Comments 


Procedures 


Images page 
(standard) 


RG62 


Note the following Hems. 

•This page displays up to eight single 
thumbnail images. 

•Images display with slngla black borders. 

•If you do not select Group Events images 
on the image Search page or you do 
not modify the default filter conditions, 
the images on the first search 
results page that displays. 


To display an enlarge single Image, 
click on a thumbnail image displayed 
on the images page. 


Images page for 
AVI mode 


FIG S3 


When a sequence Is captured In AVI mode. 
Images are saved in the following lormal: 

■The first image Is saved as a JPEG, file 

•AO images between the first image and the 
last image are saved as an AVI clip. 

•The last image is saved as a JPEG file. 


To view the AVI clip, click on 
the AVI symbol. 


Events Page 
(standard) 


FIG. 64 


Note the following items: 

•This page displays up to eight events. 

•For each event, a thumbnail image of the 
first image in the event displays. 

•Images display with black feathered 
borders thai suggest group images. 

• ir you select Group Event Images on the 

fitter conditions, the events page is the 
first search results page that displays. 
Otherwise , this page is not accessible. 


To display the expanded view of a 
specific event, click on a thumbnail 
image displayed on the events page. 


Events page 
including events 
captured in AVI 
mode 


FIG. 65 


Note the following items: 

•The AVI symbol Indicates that AccuTrack 
captured an event in AVI mode. 

•If you anticipate that your search results 
include a lot of AVI files, it fs preferable 
to de-select Group Event Images on the 
Images Search page. Then the JPEG 
files captured before and after the AVI 
clip display files on Uib first search results 
page (the images page). 


To view the AVI clip, click on the 
AVI symbol on the events page, 
and then click on the AVI symbol 
on the expanded event page. 
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Search Results Pages and Views (continued) 



Page 


Figure/ View 


Comments 


Procedures 


Expanded event 
page 




Expands an event group from the events pag 
Displays to eight thumbnail images for an 
event group selected on an events page. 


Jo display an enlarge single image, 
click on a thumbnail image displayed 
on the expanded event page. 


Single image 
page. 


FIG. 67 
FIG. 68 


Displays a single enlarge image with 
information about the imagt 


Use the following options as deslrel 

• Type comments In the Comments 
field that can be used as a filter 
condition In a subsequent 
image search. 

• Download the Image cart from 
this page 

Print Images from this page. 


Quick Viewer 
page 


FIG. 69 


Note the buttons that are enabled on the 
AccuTrack Image Control Panel when Quick 
Viewer Is selected on the Image Search pag) 


To view the search results using Qulc 
Viewer, use the AccuTrack Image 
Control Panel. 


AVI viewer 
window 


FIG. 70 


You must have a Web browser plug-In such 
as QuickTime™ to view AVI clips. [2] 


Use this window to view AVI files, 
for vour AVI viewer 


Image carl page 


FIG. 71 


Note the following items: 

This page displays thumbnail images 
for JPG files in the image cart and AVI 
symbols for AVI files in the image can 

Images display with single red borders. 




Print preview 
page 


FIG. 72 


This page displays a print preview of a 
single image. The displayed information 
also prints with the image. 




|2] You can downlot 


d QuickTime from http-//www.apple.coni/qulckllnie/. 
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Data Stored or Displayed with an Enlarged Single Image 



FIELD 


DESCRIPTION 




• SCHEDULED. Daily program sequences 

• INPUT. (Alarm) Input sequence 

• MOTION. Motion (detection) sequence 

• BLOCKEDCAM. Blocked camera sequence 

• CARD READ. Transaction sequence initialed by a can! reader at the ATM 

• PRINTER. Transaction sequence initiated by printing of ATM receipt 


Capture Time 


Date and time the image was captured, to thousandths of a second 


Image Name 


Filename of the Image on the AccuTrack hard disk 


Camera # 


Camera number (01 through 24) 


Camera Description 


Camera description entered under the Camera Setup menu option 


Image Comments 


Comments typed on a single menu page 


Alarm Description 


Alarm description entered under the Sequence Setup/input Setup menu option. 


Number of Images 


The Number of images is one of Hie following values: [1] 

• Number of images in Ihe current cache of search results 
. Number of images in the search results 

• Number of images in the selected event 


Image Number 


Sequential number of images in the Number ol images 


ATM Description 


ATM description entered under the ATM setup menu option. 


ATM Card Number 


ATM card number for the transaction 


ATM Transaction Type 


Description of the ATM activity, such as deposit, withdrawal, or card 
retained. This field may be unknown. 


ATM Print Receipt 


ATM receipt associated with the Image. This option must be configured under the 
Sequence Setup/Transaction Setup menu option. 


|1 ] On thg single Image page, the Number of Images also displays on tile Image counter. 
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AccuTrack Image Control Panel 



Buttons 

Buttons on the AccuTrack Image Search Control Panel display with one of 
the following colors: 

•White -Enabled 

• Yellow - Enabled and selected. Click to activate. 

• Gray • Disabled 

FIG. 76 




Image Counter 
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AccuTracK Image Control Panel Buttons 
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AccuTrack Image Control Panel Buttons (continued) 
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AccuTrack Image Control Panel Buttons (continued) 



IMAGES PAGE EVENTS PAGE 



fM^l[M^l[M^| 1 



Sslect boon, mimrloi. and seconds, and the then dick on Ibis button 
to go backward tee selected amoairt of time. 



visa an AVI dip from the Quick Viewer sage, navigate to the AVI symbol and 



[1| To view an AVI dip 
|2] When you perform an Image 



rca, opto 100 images ore 
ten you do not select Provl 
ul the cached Images or ei 



then click on the Launch AVI hi 
re cached to improve 
Search page, the Image 
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47 48 49 50 
51 52 53 54 



54 55 56 57 
58 59 60 61 



L_ 



55 56 57 58 
59 60 61 62 



IMAGES PAGE 



63 64 6 


5 66 


67 68 6 


9 70 



77 78 79 80 
81 82 83 84 



Note 

• Rectangles represent the images pages. 

• Numbers represent image numbers lor the thumbnail images on the images page. 

• The example shows a total ol 84 images in the search results. 
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IMAGES 
PAGE 





PRINT 

PREVIEW 

PAGE 



FIG. 82 Note 

— — ^— Rectangles represent search results pages. 



FIG. 83 



Numbers represent event numbers or image numbers. 
The example shows 5 images in event number 33. 
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474 

Image Cart Symbol 



SYMBOL / 


MAGES PAGE 


EVENTS PAGE 


EXPANDED 
EVENT PAGE 


SINGLE IMAGE 
PAGE 


SINGLE CART 
PAGE 


Block Image cart / 
symbol / 


Image It not In 
the image cart. 

Click on me black 
Image cart symbol 
to check out the 
Image (add the 

cart)!* m ° 9 * 


Click on the black 
Imoge cart symbol 
to check out aB the 
Images In tne event 
(add the Image* to 
the Image cart). [1] 


Image Is not In 
the Image cart. 

Click on the Mack 
Image cart symbol 
to check out the 
Image (add the 
Image to the Image 


■12] 


Image Is not In 
the Image can. M 

Click on the block 
Image cart symbol 
to check out the 
Image (add the 
Image to the Image 
cart). 


Red Imaga cart 

H 


Image li in 
the Image cart. 

Click on me red 
Image cart symbol 
to check In the 
Image (remove the 
Image from the 
Image carl). 


(does not display] 


Imaged In 
the Image cart. 

Click on Ihe red 
Image cart symbol 
to check In Ihe 
Image (remove the 
Image from the 
Imago cart). 


•12] 


Image Is in 

the Image cart. [3] 

Csekonthered 
Image cart symbol 
to check In Ihe 
Image (remove the 
Image from the 
image cart). 


[1] On the event* page, the color on the image earl symbol it not tlgnilicanl 

[2] The Image cart symbol doe* ml display on the tingle Image page. U the Image b hi the Image carl, 
Image Checked Out displays In red italics will) the displayed data. 

|3] On the Image cart page, all Images Initially display with the red Image cart symbols. It you click oa a red Image earl 
symbol on the Image cart page, the symbol changes to black and the Image Is removed mm the Image cart. The 
Image conlbiiiBs to display on the image cart page until you re-access the page. 



FIG. 84 



10/15/2003, EAST version: 1.04.0000 



U.S. Patent Jun. 24, 2003 Sheet 79 of 79 US 6,583,813 Bl 





AccuTrock imago Verification Viewer Downkxid 






















il I— ^ , 




FIG. 


85 



10/15/2003, EAST Version: 1.04.0000 



US 6,583,813 Bl 



SYSTEM AND METHOD FOR CAPTURING 
AND SEARCHING IMAGE DATA 
ASSOCIATED WITH TRANSACTIONS 

This application claims the benefit of U.S. Provisional s 
. Application No. 60/103,731 filed Oct. 9, 1998. 

TECHNICAL FIELD 
This invention relates to transaction record systems. Spe- 
cifically this invention relates to a transaction record system 10 
for an automated banking machine which provides image 
records as well as data records related to transactions and 
other activities. The invention further relates to a system 
which captures images and enables images and data to be 
stored, remotely accessed and selectively processed and 15 

BACKGROUND ART 

Automated banking machines are known in the prior art. 
A common type of automated banking machine used by 20 
consumers is an automated teller machine ("ATM"). ATMs 
enable customers to carry out banking transactions such as 
dispensing cash, making deposits, making transfers of funds, 
depositing checks and other instruments, payment of bills 
and accouDt balance inquiries. Other types of automated 25 
banking machines are used for purposes of dispensing 
tickets, scrip, travelers checks, airline tickets, gaming mate- 
rials and other items of value. Other types of automated 
banking machines are used by service providers such as 
cashiers or bank tellers for purposes of dispensing or receiv- 30 
ing currency, counting currency and determining the genu- 
ineness of currency. For purposes of this disclosure an 
automated banking machine will be considered as being any 
machine which accomplishes the handling or transfer of 
items having or representative of value. 

In the past some automated banking machines have 
included a camera system. Cameras have been installed 
adjacent to the machine for purposes of viewing persons 
conducting transactions. Such cameras have been connected 4Q 
to video tape recorders and record an image of the customer 
conducting the transaction. Such systems have limitations 
because the tapes used to record the images must be peri- 
odically changed. A failure to change the tape could result io 
images no longer being recorded. Changing the tape too 
soon results in a waste of available image storage space. 
Tapes that are reused wear out after a few cycles. 

A further drawback associated with tape recording sys- 
tems used in connection with automated banking machines 
is that the tapes should be maintained for a substantial 50 
period. For example if a customer claims that they did not 
conduct a transaction, this will likely not be known until 
several weeks or months after the transaction was con- 
ducted. To determine if the customer's assertions are true the 
tape must be located. This involves cataloguing and storing 55 
tapes, often for extended periods of time. This is inconve- 
nient and costly. 

A further drawback associated with conventional image 
recording systems associated with automated banking 
machines is that the camera may be blocked. Such blocking 60 
may be deliberate or inadvertent. For example a criminal 
wishing to avoid identification may cover the lens area of the 
camera so that no usable video is obtained. Alternatively, 
lighting conditions such as sun glare and shadows may 
render a camera unable to provide a suitable video image. $5 

A further limitation of existing ATM camera systems is 
that they often record only a single image during the 



transaction. If that image happens to coincide with a time 
when the customer is not facing in a direction which enables 
the camera can take a suitable picture, the image is of no 
value. Alternatively, if a camera is operated to record 
throughout every transaction the available tape supply will 
be used up more quickly. In such situations the tape supply 
is often consumed by recording images of persons with- 
drawing very small amounts from the ATM. 

A further issue that sometimes arises with regard to 
automated banking machines is shortages of materials or 
cash. Sometimes for example, the amount of cash remaining 
within a machine is less than available records suggest 
should be held therein. Similar issues may arise with other 
items that are dispensed from or deposited in automated 
banking machines. Such issues call into question whether 
there has been a machine malfunction, conversion by a 
person authorized to service the machine or theft by a third 
party. As servicers such as bank employees accessing auto- 
mated banking machines are often unsupervised, it is some- 
times impossible to determine the cause of the shortage. 

Circumstances sometimes arise in the course of transac- 
tions at automated banking machines where it would be 
desirable to inform particular individuals of the existence of 
the circumstances. Such circumstances may include for 
example that the recording media being used to record 
images needs to be changed. Other conditions may include 
that someone has gained access to the machine or an area 
surrounding the machine. It would be desirable to include an 
image with or as part of some messages. Such information 
would enable persons who need to know about such con- 
ditions to take immediate action. 

Certain types of automated banking machines also accept 
documents representative of value such as checks and trav- 
elers checks. Often issues arise concerning the authenticity 
of such items. Determining the authenticity of such items 
may be a time consuming process due to the need to 
physically retrieve such items and to compare indicia 
thereon to a genuine item. An example would be signature 
comparisons for determining whether a particular signature 
is appropriate. Such activities can be costly and time con- 
suming. 

There further exists a need for systems that operate so that 
when a triggering event occurs, the capture of images and 
other actions occur in a predetermined sequence. For 
example the occurrence of an image condition such that 
motion is sensed by a camera or that the camera is blocked, 
may cause additional images to be captured from the same 
or other cameras. In such circumstances images taken prior 
to the triggering event may be stored in correlated relation 
with the images corresponding to the event for a later 
analysis. In some circumstances it may be desirable to have 
images captured at different rates based on the nature of the . 
triggering events, including at rates which comprise gener- 
ally continuous image video capture. There further exists a 
need for systems in which a triggering event may consist of 
a change within a particular detection area, which is a subset 
of an entire image being viewed by a camera. This enables 
actions to be taken in response to changes in the detection 
area while other changes within the field of view may be 
ignored. There further exists a need to program an image 
capture system with such sequences in a simplified manner. 

There further exists a need for systems in which captured 
images can be stored and analyzed. This may include for 
example analysis by the type of triggering event which 
caused the image to be captured. Other parameters for 
searching images would also be desirable to use, such as the 
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content of the image, the type of transaction with which it is 
associated or the time frame within which the image was 
captured. The ability to search such images by one or more 
of these parameters would greatly reduce the time necessary 
to locate desired images. In addition there exists a need for 
the selective deletion of images when available storage 
space approaches depletion. Such selective deletion may 
include eliminating image data associated with certain types 
of image events while selectively retaining other informa- 

There further exists a need for a system which can provide 
increased assurance of the authenticity and unaltered con- 
dition of an image. In some circumstances there may be 
concern that a captured image has been altered so that it is 
different from its original form. The ability to document that 15 
images are unaltered despite transfer from an image capture 
system to another system increases the evidentiary value of 
the captured image. 

There further exists a need for an image capture system 
that can be used in connection with automated transaction 20 
machines as well as in other transaction or service environ- 
ments. Such a system may record activities and transactions 
occurring within facilities at particular times to enable 
documentation of events that occur. 

Thus there exists a need for a transaction record system 25 
which includes image capture capabilities that overcome the 
limitations associated with existing systems. 

DISCLOSURE OF INVENTION 

It is an object of the present invention to provide a 30 
transaction record system for an automated banking 
machine. 

It is a further object of the present invention to provide a 
system for recording image data related to transactions 
conducted at an automated banking machine. 

It is a further object of the present invention to provide a 
system for recording image and transaction data related to 
transactions conducted at an automated banking machine. 

It is a further object of the present invention to provide a 
system for enabling access to image and transaction data 
related to transactions conducted at an automated banking 
machine remotely through a network. 

It is a further object of the present invention to provide a 
system for accessing image and transaction data in response 
to events that occur at an automated banking machine. 

It is a further object of the present invention to provide a 
system for obtaining image data and for taking actions in an 
appropriate programmed sequence in response to events at 
an automated banking machine. 

It is a further object of the present invention to provide a 
system for obtaining image data and for taking actions in a 
sequence which can be programmed more readily. 

It is a further object of the present invention to provide a 
system which achieves greater assurance of capturing useful 
image and transaction data related to transactions conducted 
at an automated banking machine. 

It is a further object of the present invention to provide a 
system that provides selective remote notification of events 
and conditions occurring at an automated banking machine. 

It is a further object of the present invention to provide a 
system that simplifies the recording, storage and analysis of 
transaction and image data generated at an automated bank- 
ing machine. 

It is a further object of the present invention to provide a 
system for imaging documents at an automated banking 



It is a further object of the present invention to provide a 
system that enables analysis of documents presented at an 
automated banking machine. 

It is a further object of the present invention to provide a 
5 system for obtaining and storing image data related to 
servicing activities conducted at an automated banking 
machine. 

It is a further object of the present invention to provide 
methods of recording, storing and analyzing image and 
10 transaction data produced in the operation of an automated 
banking machine. 

It is a further object of the present invention to provide an 
image capture system that captures images in response to 
triggering events and which captures images in accordance 
15 with a programmed sequence of actions. 

It is a further object of the present invention to provide an 
image capture system in which the triggering event is the 
occurrence of an image condition within a field of view of 
a camera. 

It is a further object of the present invention to provide an 
image capture system in which images are captured in 
response to a triggering event which includes changes within 
a selected detection area within a field of view of a camera. 
It is a further object of the present invention to provide an 
5 image capture system which captures images in response to 
a triggering event which includes the presence of a particular 
characteristic, object, facial feature or color within an image. 

It is a further object of the present invention to provide an 
image capture system that can provide remote notification of 
a triggering event and to include with the notification an 
image associated with the event. 

It is a further object of the present invention to provide an 
image capture system which provides for selective deletion 
. of images by the type of event with which the image is 
associated. 

It is a further object of the present invention to provide an 
image capture system in which images may be sorted and 
searched by parameters or combinations of parameters, 
i It is a further object of the present invention to provide an 
image capture system which provides for transferring 
images in a manner which provides enhanced assurance that 
the transferred images have not been altered. 

Further objects of the present invention will be made 
45 apparent in the following Best Modes For Carrying Out 
Invention and the appended claims. 

The foregoing objects are accomplished in one exemplary 
form of the present invention by a system which includes an 
automated banking machine. The automated banking 
50 machine carries out transactions by performing several 
transaction functions. At least one camera is positioned 
adjacent to the automated banking machine. The camera 
operates to produce camera signals which represent images 
within a field of view of the camera. 
55 The system further preferably includes a computer in 
operative connection with a data store. The computer 
includes a server operating in connection with the computer. 
In some embodiments of the system the server may be 
resident and a part of the computer within the automated 
60 banking machine. In other embodiments the server may 
reside in a location adjacent to or disposed from the auto- 
mated banking machine. The computer is in operative con- 
nection with the machine and the camera. The computer 
operates in accordance with programmed instructions to 
65 include image data corresponding to the camera signals in 
the data store. The image data is stored in response to the 
machine carrying out transaction functions. 
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n with an electronic communi- In certain exemplary embodiments the triggering events 

is network. In some embodiments this may be a direct include certain image conditions. Image conditions may 

:ction, local area network or an intranet. In other correspond to the blocking of a camera in a way that 

embodiments the network may be a telecommunications prevents the delivery of usable video. Alternatively image 

network, wide area network or the Internet. Alternatively the s conditions may correspond to the detection of motion, 

network could include a wireless network such as an RF colors> objcctJ , fadal fcatu clolMn ^ i(i Q 

network or satellite network. A user terminal disposed from other characteristics or items * fiel / 0 f ^ 0 ' f a 

die automated banking machine .s connected to the network^ camera Exemplary embodiments of the invention enable a 

The user terminal may be connected directly to the network user to sdec , ^ Qf mQK sllbse(s of , he fie]d f ^ 

nl^IX, H° 0nnCCte ^ ^ ?• Ta mterVem , ng 10 detection area and to sense for motion or other image 
networks and servers. The user term.nd includes a computer conditions only me detection area whik { > 
with a browser operating therein as well as an output device ^ conditions outsidc the deteclion area 
such as a screen or pnnter. The user terminal through the me detection of desired events ^ avoids ^ f ^. 
browser accesses die image data through the server. The user able ima st jn ^ tQ ^ . whjcn 
termuial operates to output the unage data through its output 15 are not Q f mterest 
device. The user terminal also is used to access transaction „ , ,' . . 
data related to transactions corresponding to image data . E^nrplary embodiments of the mvenhon also store unage 
stored in the data store associatedwith the server. The user da f a other data associated therewith so as to.provide 
terminal may be operated to more rapidly analyze and sort cnhanced searchability of images. Embodiments of the 
image and transaction data, as well as to conduct a more „ mvenl1011 enable searching through images selectively by 
detailed analysis of image or transaction data one or morc Parameters. s « ch parameters may include the 
Embodiments of the system also operate in accordance ^ ° f eVent ^ I"!? 8 ? * be , Cap f ed ' 
• with sequences of instructions. The sequence instructions 6 h Xemplaly P^ers ™y "iclude transaction types 
provide for a sequence of actions to be taken in response to , Whlcb an image , 15 TnT ' * T T 
certain conditions. For example if in the course of capturing a which f imagCS arC a P b ^f Other parameters for the search- 
image data, a camera adjacent to the ATM is unable to * ^ ° f ™«B« delude searcmng by color, object ype, 
produce usable video, such as because it is covered or due ^ ° °^r character^tics Such capabilities 
to glare, the instructions in the sequence may cause the "f 1 ? imag6S t0 . b f. ldea }^ d > ™f°* 
system to begin capturing image data from another nearby ShS^tof 3 ' " " * 
camera. Other sequence instructions may avoid recording 30 y 

the image data for selected transactions. Other sequence Exemplary embodiments of the present invention also 

instructions may cause the recording of images associated enable ^ transfer of images that have been captured in a 

with service activities. This may be done with cameras manner that provides greater assurance that the images have 

located in service areas such as behind an automated teller not been sub i ect to alteration. Such capability assures that 

machine or within the automated teller machine itself. 35 the image has greater evidentiary value in the event that it is 

Selective motion detection and other hard and soft triggers used to estlb lish liability related to the a 

may be used to initiate recording of images and/or other actions or events, 
actions which are part of a sequence. 

Alternative embodiments of the invention also provide 

messages through the network indicative of conditions or to FIG. 1 is a schematic view of an exemplary embodiment 

events occurring at the automated banking machine. Further of a transaction record system of the invention used in 

embodiments of the system operate to manage available connection with an automated banking machine, 

memory. This may include projecting when available FIG. 2 is a schematic view of a control system for devices 

memory will likely be depleted and sending a message within an automated banking machine which incorporates a 

which indicates such condition. Other embodiments may 45 first embodiment of a transaction record system of the 

reconfigure available memory or may automatically delete invention. 

selectively certain image data or off-load data in memory FIGS 3^4^ schematic vi ews of ^ relationship 

through the network to a remote storage locaUoa Further belwecn th(J , ; ca] ^ nta which makc cnjh J_ 

alternative embodiments of die invention provide for imag- men(s of ^ transactioD record stem of the in £ enlion . 

mg of documents deposited in the machine. Image and 50 _ . , , , . , 

transaction data associated with the deposit of documents FIG " 5 15 a schematic view of the operation of the logical 

may be recovered. Analysis of document image data such as components of an embodiment of the invention operating to 

signature analysis may be conducted through the network delect motlon in field of wew of a camera ^ 
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remotely from document verification terminals which conn ection with an embodiment of the ir 

include data usable to verify the genuineness of deposited 55 FIG - 6 K a schematic view of the operation of logical 

documents. components of an embodiment of the invention responding 

Alternative exemplary embodiments of the invention are t0 a hard tri 8g er W m P ut - 

used independent of an automated transaction machine to 7 is a schematic view of the logical components of 

capture image data selectively in response to triggering an embodiment of the invention responding to a soft trigger 

events. Such triggering evcQts operate to cause the system to 60 mP 1 ". 

execute sequences which may include the capture of addi- FIG. 8 is a schematic view of the operation of the logical 
tional images or to take other actions including the remote components of an embodiment of the invention responding 
notification of persons electronically of the occurrence of to loss of usable video from a camera, 
triggering events. Such notifications may include or have FIG. 9 is a schematic view of the logical components of 
attached thereto at least one image file corresponding to an 65 an embodiment of the present invention operating in con- 
image associated with the event for which notification is nection with a user identification system which identifies a 
being given. user based on visible properties associated with the user. 
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FIG. 10 is a schematic view of an alternative embodiment 
of a transaction record system of the present invention in 
which an image server resides with other servers which 
operate the automated banking machine. 

FIG. 11 is a schematic view of a further alternative s 
embodiment of a transaction record system of the present 
invention in which the image acquisition devices are sepa- 
rate nodes on a network. 

FIG. 12 is a schematic view of a further alternative 
embodiment of the transaction record system of the present 10 
invention in which the image acquisition devices reside in a 
second network. 

FIG. 13 is a schematic view of a further alternative 
embodiment of the present invention which includes an 
automated banking machine with a document imaging 
device. 

FIGS. 14 and IS are a schematic view of logic flow 
associated with memory allocation and control used by 
embodiments of the present invention to provide greater 
reliability in storing image and transaction data. 

FIG. 16 is a screen presented in an exemplary embodi- 
ment of the invention at a user terminal describing functions 
performed by an exemplary system of the invention and 
categories of persons generally authorized to perform such 
functions. 

FIG. 17 is a exemplary embodiment of a screen presented 25 
at a user terminal for purposes of viewing and analyzing 
image data. 

FIG. 18 is a screen presented at a user terminal in an 
exemplary embodiment of the invention for purposes of 
explaining the functions of icons shown in FIG. 17. 30 

FIG. 19 is a view of an exemplary screen similar to FIG. 
17 but with a selected image enlarged for purposes of 
analysis. 

FIG. 20 is a view of an exemplary programming screeD 
used in an embodiment of the invention. 35 

FIG. 21 is an exemplary embodiment of a daily program 
screen presented at a user terminal in an embodiment of the 
invention. 

FIG. 22 is an exemplary embodiment of a setup screen 
displayed at a user terminal. 40 

FIG. 23 is an exemplary embodiment of a setup screen 
presented at a user terminal for purposes of setting image 
compression types and for programming sequences. 

FIG. 24 is an exemplary embodiment of a screen pre- 
sented at a user terminal for purposes of establishing user 4 
access capabilities. 

FIG. 25 is an exemplary screen presented at a user 
terminal for purposes of establishing image and data capture 
parameters during the carrying out of transaction functions J0 
at an automated banking machine. 

FIG. 26 is an exemplary embodiment of a screen pre- 
sented at a user terminal for purposes of input and editing 
e-mail addresses used for sending messages related to con- 
ditions and events occurring at an automated banking 55 
machine. 

FIG. 27 is an exemplary embodiment of a screen pre- 
sented at a user terminal for purposes of setting up an e-mail 
group including e-mail addresses of persons to be notified in 
response to the occurrence of conditions and events at an 60 
automated banking machine. 

FIG. 28 is a schematic view of an alternative embodiment 
of an image capture system of the present invention. 

FIG. 29 is an exemplary screen presented at a user 
terminal for purposes of operating and controlling the cap- 65 
ture and presentation of captured images in the system of 
FIG. 28. 



FIG. 30 is a detailed view of the tool bar and icons 
presented in the screen shown in FIG. 29. 

FIGS. 31 and 32 are a chart showing the icons presented 
in the tool bar shown in FIG. 30 and the functions and 
operations in the programming of the exemplary system 
associated with each. 

FIG. 33 is an exemplary screen presented to a user in 
operation of the exemplary system shown in FIG. 28 for 
purposes of configuring the selective deletion of image data. 

FIG. 34 is an exemplary screen presented at a user 
terminal in the system of FIG. 28 for purposes of setting up 
an automatic delete function for selectively deleting types of 
captured images. 

FIG. 35 is an exemplary screen presented at a user 
terminal for configuring and programming the exemplary 
system to apply enhanced security to captured images. 

FIG. 36 is an exemplary screen presented at a user 
terminal for purposes of applying descriptive names to 
cameras, which descriptive names may be used in program- 
ming sequences. 

FIG. 37 is an exemplary screen presented at a user 
terminal which enables a user to assign descriptive names to 
outputs which may be provided by the system as part of 
sequences. 

FIG. 38 is an exemplary screen presented at a user 
terminal which enables a user to assign descriptive names to 
inputs which the image capture system may receive. 

FIG. 39 is an exemplary embodiment of a screen pre- 
sented at a user terminal for purposes of capturing images in 
response to triggering events which occur in the operation of 
an automated banking machine. 

FIG. 40 is an exemplary screen presented at a user 
terminal for purposes of establishing e-mail addresses and 
groups of e-mail addresses which are to receive e-mail 
messages in response to the occurrence of certain triggering 
events in the system. 

FIG. 41 is an exemplary embodiment of a screen pre- 
sented at a user terminal for purposes of setting up a group 
of e-mail addresses for persons who are to be notified of 
certain events occurring at the system. 

FIG. 42 is an exemplary embodiment of a screen pre- 
sented at the user terminal for purposes of programming the 
system with sequences. 

FIG. 43 is an exemplary screen presented at a user 
terminal which graphically displays sequences applicable to 
particular times and dates that have been programmed into 
the system. 

FIG. 44 is an exemplary screen presented at a user 
terminal showing the times each day certain routine program 
sequences are carried out. 

FIG. 45 is an exemplary embodiment of a screen pre- 
sented at a user terminal representative of the steps taken by 
a user in programming a sequence. 

FIG. 46 is an exemplary embodiment of a screen pre- 
sented at the user terminal for purposes of establishing a 
programmed sequence in response to inputs received by the 
system. 

FIG. 47 is an exemplary screen presented at a user 
terminal for purposes of displaying the times during which 
the sequence applicable to a particular input will cause a 
system to operate. 

FIG. 48 is an exemplary screen presented at a user 
terminal associated with programming a sequence in 
response to receipt of a particular input by the system. 
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FIG. 49 is an exemplary screen presented at a user 
terminal for purposes of configuring a sequence for captur- 
ing images in response to detection of motion. 

FIG. 50 is an exemplary screen presented at a user 
terminal for purposes of establishing a detection area as a 
subset of a field of view of a camera for purposes of 
selectively detecting motion within the detection area. 

FIG. 51 is an exemplary screen presented at a user 
terminal for purposes of showing when a sequence appli- 
cable to detection of motion will be operative within the 
system. 

FIG. 52 is an exemplary screen presented at a user 
terminal for purposes of programming a sequence to be 
carried out in response to detection of a motion event. 

FIG. 53 is an exemplary screen presented at a user 
terminal associated with programming a sequence for 
detecting lack of usable video from a camera in which a 
camera is selected. 

FIG. 54 is a screen similar to that in FIG. 53 showing how 
the screen after a camera is selected in response to presen- 
tation the screen shown in FIG. 53. 

FIG. 55 is an exemplary screen presented at a user 
terminal for enabling a user to select a degree of change in 
an image for purposes of detecting motion in an image. 

FIG. 56 is an exemplary screen presented at a user 
terminal indicative of when a particular motion detection 
sequence will be executed by the system. 

FIG. 57 is an exemplary screen presented at a user 
terminal for purposes of programming a sequence to be 30 
executed in response to a lack of usable video condition. 

FIG. 58 is an exemplary screen presented at a user 
terminal for purposes of establishing a sequence for captur- 
ing images at an automated banking machine. 

FIG. 59 is an exemplary screen for establishing a 35 
sequence for capturing images in connection with a particu- 
lar type of transaction and enabling a user to selectively 
input times at which images will be captured as well as the 
rate of image capture. 

FIG. 60 is an exemplary embodiment of a screen pre- ao 
sented at the user terminal for purposes of programming a 
sequence and demonstrating the capability of a user to 
establish the image capture rates as well as the image quality 
associated with storage of captured images. 

FIG. 61 is an exemplary embodiment of a screen pre- 45 
sented at a user terminal for purposes of a user selecting the 
recovery of images by various parameters. 



10 



FIG. 68 is an exemplary screen similar to FIG. 67 in 
which the presented indicia indicates that the image has been 
grouped with a particular event. 
FIG. 69 is an exemplary embodiment of a screen pre- 
5 sented at a user terminal in response to search results 
obtained in response to a quick viewer routine in which a 
user is enabled to navigate through images by selecting 
buttons on the control paneL 
lQ FIG. 70 is an exemplary embodiment of a screen pre- 
sented at the user terminal of a quick viewer page showing 
a single image with the selected image in enlarged format. 

FIG. 71 is an exemplary embodiment of a screen pre- 
sented on a user terminal in which a user is enabled to view 
J5 images. 

FIG. 72 is an exemplary embodiment of a screen pre- 
sented at a user terminal which displays images selected for 
purposes of preview for printing or transfer in an "image 
cart" which enables such images to be downloaded. 
20 FIGS. 73 and 74 are a chart indicating the features 
associated with the different search results shown in FIGS. 
62 through 72 and the features and capabilities of the images 
associated therewith. 

FIG. 75 includes a chart of indicia and information 
25 displayed with images which can be searched in the exem- 
plary embodiment of the invention. 

FIG. 76 is an exemplary embodiment of the control panel 
displayed on screens of a user terminal in connection with 
the presentation of search results. 

FIG. 77 is an exemplary embodiment of an image counter 
presented in connection with the control panel shown in 
FIG. 76. 

FIGS. 78 through 80 are charts showing the various 
functions performed by selection of icons in the exemplary 
control panel when particular image pages are being dis- 
played. 

FIGS. 81 through 83 are schematic views showing the 
operation of the icons included in the exemplary control 
panel screen in navigating through images which are pre- 
sented to a user at a user terminal. 

FIG. 84 is a chart explaining variations in an icon used in 
connection with designating images for deposit into an 
image cart for purposes of downloading images as a group, 
and the functions associated with the icon. 

FIG. 85 is an exemplary embodiment of a screen pre- 
sented at a user terminal for purposes of providing the user 
with greater image integrity assurance for downloaded 



FIG. 62 is an exemplary screen presented at a u^i , . , 

terminal showing icons presented as a control panel and ^ * T iqUe key 01 P 358 ^ for Ptoses of 

images recovered in response to a search. 50 enabllD S tbe untocta »g ° f such ^ages. 

FIG. 63 is a view of the screen similar to FIG. 62 but BEST MODES FOR CARRYING OUT 

including representations of images captured as continuous INVENTION 
video in AVI form. n , . 

FIG. 64 is an exemplary embodiment of a screen pre- « ^1° draW,DgS ? d P^" 1 " 1 * <° ™S. 1 

lin^nn^^^^^.-^^.VKV. ss mere 15 "> exemplary embodiment of the 

present invention which operates as a transaction record 
system for an automated banking machine generally indi- 
cated 10. The system of this embodiment includes an 
automated banking machine 12 which in this example is an 
ATM. It should be understood that in other embodiments of 
the invention other types of automated banking machines 
may be used. ATM 12 includes a number of transaction 
function devices. These transaction function devices are 
associated with components of the machine such as a card 
iplary embodiment of a screen pre- 65 reader 14 and a keypad 16. The card reader and keypad serve 
terminal showing an image output in which as input devices through which users can input instructions 
tv P e - and information. It should be understood that as referred to 



sented at a user terminal in response to a search in which the 
search results show that a plurality of images have been 
captured in response to a triggering event. 

FIG. 65 is a view of a screen similar to FIG. 64 including 
representations that images have been captured as continu- 
ous video in response to certain triggering events. 

FIG. 66 is an exemplary embodiment of a screen pre- 
sented at the user terminal showing a plurality of images 
captured in response to a single triggering event. 

FIG. 67 is an 
sented at 

images are not grouped by particular e> 
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herein the keypad includes function keys or touch screen 
inputs which may be used in other embodiments to input 
data into the machine. 

ATM 12 further includes additional transaaion function 
devices. Such transaction function devices may include a 5 
presenter schematically indicated 18 which operates to 
present cash or other documents of value to a customer. The 
presenter 18 in the embodiment shown is associated with a 
dispenser schematically indicated 20 (see FIG. 2). The 
dispenser is operative to obtain sheets such as currency bills it> 
from within the machine and to deliver them to the presenter 
in the described embodiment. In alternative embodiments 
only a presenter or a dispenser may be used. The exemplary 
AIM 12 further includes a depository 22. The depository 22 
accepts deposits from customers. In the embodiment shown 15 



the depository is generally configured to accept cash and 
other instruments such as checks from a customer. It should 
be understood that in other embodiments other types of 
depositories which accept various types of items represen- 
tative of value may be used. 2 

The transaction record system of the described embodi- 
ment further includes a first camera 24. Camera 24 is 
positioned within or behind the fascia of the ATM or 
otherwise adjacent the ATM so as to have a field of view 
which generally includes the face of the user operating the 2 
ATM. Afurther camera 26 is positioned adjacent to the ATM 
and includes a field of view which includes a profile or other 
view of the user operating the ATM. 

A further camera 28 in this exemplary system is shown 3 
positioned adjacent to the ATM with a field of view to 
observe a service area of the ATM. Camera 28 in the 
exemplary embodiment shown is directed to observe the 
back of the ATM and is usable for observing or detecting 
service activities. Camera 28 may be for example positioned 3 
within a vestibule or room which is accessed by service 
personnel for purposes of servicing the ATM. A further 
camera 30 shown schematically, is positioned adjacent the 
ATM and within the interior of the cabinet of the ATM. 
Camera 30 is shown having a field of view which is directed 
generally opposite to that of camera 28 and enables it to 
view areas which would normally include the face and hands 
of servicing personnel. Camera 30 preferably operates when 
a service door 32 is open and a servicer is accessing the 
interior of the machine. This enables capturing image data 4 
related to persons servicing or accessing the interior of the 
machine. 

In the embodiment shown each of the cameras 24, 26, 28, 
30 provides camera signals which are analog signals repre- 
sentative of what is observed within the field of view of the 5 i 
respective camera. It should be understood that the camera 
configuration shown in FIG. 1 is exemplary and other 
configurations of cameras, or greater or lesser numbers of 
cameras, may be used in connection with embodiments of 
the invention. It should further be understood that embodi- 5 . 
merits of the invention may include digital cameras or other 
types of devices from which images may be reproduced. 

FIG. 2 shows a schematic view of a first hardware 
configuration of a transaction record system of the inven- 
tion. The automated banking machine 12 includes the trans- 6< 
action function devices 14, 16, 18, 20, 22 which communi- 
cate through and are operated responsive to signals passed 
through device interfaces 34. The device interfaces commu- 
nicate with the transaction function devices on an interface 
bus 36. The messages which control operation of the various 6'. 
transaction function devices are communicated through the 
interface bus. A computer which is referred to as a terminal 



In the embodiment shown in FIG. 2 an image recorder 
device 40 is shown connected to the interface bus 36. Image 
recorder device 40 in the embodiment shown is a separate 
hardware component from the automated banking machine. 
Image recorder device 40 includes a computer which 
includes a server operating therein, and further includes a 
data store schematically indicated 42. The data store holds 
programmed instructions. The data store also holds data 
representative of image data, transaction data and other data 
as later described. It should be understood that although a 
data store within the image recorder device is described in 
the exemplary embodiment, reference to a data store herein 
encompasses either a single data store or a plurality of 
connected data stores from which data may be recovered. 

Image recorder 40 receives the analog signals from the 
connected cameras 24, 26, 28 and 30 as shown. It shoutd be 
understood that embodiments of the invention may include 
devices which in addition to image data, acquire sound data, 
infrared signal data and other types of data which can be 
sensed by sensing devices, stored, recovered and analyzed 
by the system. Image recorder device 40 further includes 
inputs which are schematically represented as hard and soft 
triggers. Hard triggers, examples of which are hereinafter 
described, are signals from "hard devices" such as sensors. 
Such devices can generally sense actions or conditions 
directly such as that a service door on the ATM or to a 
service area has been opened. The image recorder device 
also receives soft triggers which may include signals repre- 
sentative of conditions or instructions which are being sent 
as signals to other devices. Such soft triggers may further 
include the signals on the interface bus 36 in the embodi- 
ment shown or timing signals or other signals usable to 
operate the image recorder responsive to programmed 
instructions, time parameters or other conditions or signals. 

Soft triggers may also include timing functions. In some 
embodiments the image recorder may monitor other types of 
transaction messages and may operate in response thereto. 
Such alternatives may include for example, systems where 
the image recorder device 40 is not connected to the bus with 
the transaction function devices, but instead monitors trans- 
action messages being sent between an automated banking 
machine or other device and a remote computer, and extracts 
information concerning the operation of transaction function 
devices from such messages. Other configurations and 
operational capabilities of the image recorder device will be 
apparent to those skilled in the art from the description 

Image recorder 40 in the exemplary embodiment is in 
communication with an electronic communications network 
schematically indicated 44. Network 44 in the described 
embodiment may be a local area network such as an intranet 
or may be a wide area network such as the Internet. In the 
embodiment shown network 44 is a network that commu- 
nicates messages in protocols such as TCP/IP. The network 
is used.to further communicate HTTP messages including 
records such as HTML, XML and other markup language 
documents. Of course in other embodiments of the invention 
other communications methods may be used. 

The image recorder device 40 includes a computer oper- 
ating at least one server. The server is connected to the 
network and has at least one uniform resource locator (URL) 
or other system address. This enables the server to be 
accessed by other terminals connected to the network as well 
as to selectively deliver messages to connected terminals. It 
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should be understood that network 44 may be connected 
through intermediate servers to other networks. This enables 
the image recorder device 40 to communicate with other 
types of remote terminals including terminals connected to 
wireless interfaces such as pagers and cellular phones. If 
network 44 is an intranet, intermediate servers which operate 
as a firewall may be included in the system. Access to the 
Internet enables the communication of messages to termi- 
nals located anywhere in the world. Such communications 
capability may be valuable in embodiments of the invention 
for purposes of image and transaction data recovery and 
analysis, and for purposes of sending messages to individu- 
als to be notified of conditions which exist at the automated 
banking machine. 

A plurality of terminals 46 are shown connected to the 
network 44. Terminals 46 may include a user terminal for 
purposes of programming parameters into the data store 42 
of image recorder device 40. Alternatively terminals 46 may 
include user terminals which may be used to analyze and 
recover image data and transaction data from the image 
recorder device. Alternative terminals 46 may include data 
stores for storing image and transaction data which is 
downloaded from the image recorder device for purposes of 
storage as later described herein. Alternative terminals 46 
may include document verification terminals for verifying 
the authenticity of documents, identifying user data or for 
carrying out other functions described herein. Typically 
terminals 46 include computers including a browser com- 
ponent schematically indicated 48. The browser communi- 
cates with the server in the image recorder device to access 
the image data. Such a browser component may be com- 
mercial browsers such as Netscape Navigator™, Microsoft 
■ Internet Explorer™ or other types of browsers. Terminals 46 
also include other software and hardware components sche- 
matically indicated 50 suitable for processing image data, 
transaction data and other data that may be obtained by 
accessing the server in the image recorder device 40. 

An exemplary terminal indicated 52 is shown in greater 
detail in FIG. 2. Exemplary terminal 52 may be 
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hardware used in carrying out the described functions. As 
shown in FIG. 3 the input signals from the cameras, micro- 
phones or other input devices are input to the device 
switching controller component 66. The device switching 

S controller component in embodiments of the invention may 
include several components. The switching controller deliv- 
ers signals, which in the described exemplary embodiment 
are analog signals, selectively in response to a record 
acquisition control component 68. The record acquisition 68 

10 component receives hard and soft trigger signals including 
signals which control or otherwise indicate the operation of 
the transaction function devices in the automated banking 
machine or other signals which arc used as an indicator to 
initiate a sequence of actions. The record acquisition com- 

15 ponent executes the instructions which indicate which image 
signals are desirable to process and record in response to the 
trigger signals. The record acquisition component further 
includes or works in connection with stored instructions, 
which are operative to detect conditions such as loss of 

20 usable video from a camera or other input device, and to 
begin acquisition of data from other devices in response 
thereto. 

The exemplary record acquisition component also oper- 
ates in connection with stored programmed instructions to 

25 sense motion in the field of view of selected cameras or other 
input devices. As later described such instructions may 
include limiting the area of analysis to one or more selected 
detection areas within a field of view, and disregarding other 
areas. The record acquisition component may further pro- 

30 cess and pass off other data such as transaction data related 
to the operation of an automated banking machine for 
storage in correlated relation with image data. In some 
embodiments of the invention transaction and other numeri- 
cal type data is selectively captured and stored in file records 

35 that are maintained separately from image data. Such trans- 
action data may be correlated with image data at the time 
(which also indicates a date or other period of time) asso- 
ciated with the activity which is recorded for both image and 
transaction data. However in other embodiments of the 



terminal, document verification terminal, data storage 40 invention other methods for such correlation may be used. 



terminal, data analysis terminal or other type of terminal fc 
inputting instructions or analyzing data available in the 
system. Terminal 52 in the exemplary embodiment includes 
a computer schematically indicated 54 which includes an 
associated data store schematically indicated 56. As with 4 
other data stores described herein, data store 56 may be a 
single data store or a number of operatively connected data 
stores. Terminal 52 further includes in operative connection 
with the computer 54, input devices 58 and 60 which include 
a keyboard and mouse respectively in the embodiment 5 
shown. Of course in other embodiments other types of input 
devices may be used. Terminal 52 further includes output 
devices. The output devices in the embodiment shown 
include a monitor with a display 62 and a printer device 64. 
Of course in other types of terminals other types of output s 
devices may be used. The terminal 52 includes a computer 
with a browser component as previously described. The 
browser in the terminal communicates with the server in the 
image recorder device 40 through the network 44 for pur- 



In this exemplary embodiment the record acquisition 
component in accordance with programmed instructions 
further controls encryption techniques used in connection 
with image data, as well as data compression techniques 
which are used for storing images. The record acquisition 
component may further operate to store data and control 
other activities such as the sending of e-mail or other 
messages in response to the occurrence of certain conditions. 

The record acquisition component 68 in this embodiment 
operates to send one or more camera signals to a frame 
grabber component 70. The frame grabber component is 
operative to generate digital image data corresponding to the 
analog camera signals which are passed to the frame grabber 
by the record acquisition component. Of course in embodi- 
ments where digital cameras are used the image data does 
not need to be digitized by a separate component. The image 
data from the frame grabber in this exemplary embodiment 
is passed to an encryption/authenticate component 72 which 
may be operated to include authenticating information 



poses of carrying out the functions later described in detail 60 within the image data. Such authentication data may include 

herein. Terminal 52 may also have a server operating therein digital signatures, digital watermarks or other data which 

as well as other software components. can be used to verify that an image has not been tampered 

The operation of exemplary embodiments of the invention with since it was acquired. In addition component 72 may 

are further described with regard to the interaction of logical operate to encrypt image data so as to minimize the risk of 

components of the system described in connection with 65 such data being accessed by unauthorized persons. In alter- 

FIGS. 3 through 9. It should be understood that the logical native forms of the invention such an encryption component 

components are generally combinations of software and may not be used. 
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A data compression component 74 may operate to com- 
press the image data to minimize the amount of storage 
required for holding it. Such data compression may be 
performed through a number of different standard or non- 
standard schemes. The degree of data compression may be . 
selectively controlled. In this exemplary embodiment of the 
invention, the degree of data compression is programmable 
and may be changed through real time inputs or may be 
programmably controlled to change the degree of data 
compression. For example instructions stored in connection 
with the record acquisition component 68 may dictate that in 1 
response to certain events which are detected through hard 
or soft triggers, high quality image acquisition is required. In 
such cases data compression may not be used or a lesser 
degree of data compression may be used, to increase the 
quality of the images. Of course in such circumstances the 1 
record acquisition component may also increase the fre- 
quency at which images are captured from various input 
devices. In some instances, the image capture frequency 
may be increased to the extent that clips of generally visually 
continuous images are captured and stored. 

After the image data is compressed it is transferred to a 
RAM cache store component 76. The RAM cache store 
stores the image and transaction data (and other system data 
that the record acquisition component may dictate be stored 
for a period of time). It should be understood that embodi- 
ments of the invention may operate to analyze cache store 
data for purposes of detecting and analyzing image and 
transaction data and for taking action in response thereto in 
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access through an intranet schematically indicated 86. As 
previously mentioned, terminals connected to the intranet 
may be used to access the stored data. A server 88 which 
operates as a firewall may be used to provide selective 
access to the intranet and to provide access to other net- 
works. Such other connected networks may include a wide 
area network such as the Internet. 

Alternatively an interface 90 may be used to provide 
access directly to the Internet schematically indicated 92. 
Appropriate controls may be used to minimize the risk of 
unauthorized access such as passwords and/or public key 
encryption. Digital signatures, session keys and the like may 
also be used to limit access to authorized persons. 

An interface 94 may be provided to telephone commu- 
nications networks. This may be accomplished through a 
dial up connection or a cellular connection. Such an inter- 
face may be provided for purposes of sending messages such 
as pager, fax or voice mail communications selectively to 
remote users or facilities. 

An interface 96 to a lease line or other dedicated com- 
munications line may be provided for purposes of providing 
for both messaging and data communication. Of course in 
other embodiments other types of communications inter- 
faces for communicating messages and for providing access 
to image and transaction data may be used. The particular 
configuration used will depend on the needs of the system 
and the capabilities of the remote communications method. 
As discussed previously, the file management component 



accordance with programmed instructions. In some cmbodi- 30 82 may be in operative connection with aloxed local storage 

ments of the invention the record acquisition control com- component such as a data store schematically indicated 98 

ponent 68 operates to place images in storage from all The local data store 98 may in the embodiments of the 

cameras on a regular or nonregular periodic basis. These invention include database software operating in a data store 

records initially do not correspond to any triggering event. in connection with a processor or computer in the automated 

However, embodiments of the invention may operate in 35 banking machine. Alternatively the database may operate on 

response to programmed instructions when a triggering the computer within the image recorder device 40 or in other 

event occurs to associate one or more images immediately computers operatively connected with the image recorder 

proceeding the triggering event to be associated with the device. 

images captured in response to the triggering event. This [ n embodiments of the invention, the image recorder 

enables embodiments of the system to capture and retain 40 device 40 or a connected device may include an image and 

those images of conditions which exuded prior to an event. transaction data recorder schematically indicated 100 in 

Such rmages may often provide valuable information con- FIG. 4. The transaction data recorder operates to record 

cerning activities that preceded and/or caused the event. image and/or transaction, or other data on a removable 

In this exemplary embodiment, from the RAM cache storage medium 102, such as a CD-R/W or other storage 

store image and transaction data is transferred in the system 45 device. Such a removable storage device may include a 

to a disk cache store 78. From the disk cache store 78, image permanent storage media which requires periodic 

and transaction data is subsequently transferred to an archive replacement, but which is not subject to later possible 

store component 80. The archive store component may in modification as is the case with erasable storage. Such 

some embodiments be a permanent or temporary storage removable storage media may work-in conjunction with 

media such as a removable storage media as hereinafter J0 other local storage or remote storage Operating under the 

described. Alternatively the archive store disk may be a control Of the file management component 82, this feature 

CD-R/W type device or similar storage media which may may in embodiments of the invention enable storage of data 

provide temporary or permanent non-modifiable storage of in other data stores which accepts overflow data on a 

image and/or transaction data. Alternatively various types of temporary basis when the removable storage medium has 

storage devices that may be off loaded or overwritten may be 5S become filled. When the removable storage media is 

changed, the recorded data in temporary storage in the other 



The archive store component operates in connection with 
a file management component 82. The file management 
component 82 operates in accordance with programmed 
instructions to perform various operations. The file manage- 
ment component works in connection with other compo- 
nents to provide access to stored image and transaction data. 
The file management component also enables control of 
available memory to facilitate storage of data and minimize 
the risk that transaction and image data will be lost. 

As represented in FIG. 4 the file management component 
82 may work in connection with interface 84 to provide 



data store is transferred thereto. Alternatively, the file man- 
agement control component may operate to periodically 
erase images and data as storage space is needed. This may 
60 be done selectively based on the age of the image, the nature 
of the event causing image capture or other parameters. Of 
course other approaches may be used. 

As previously discussed, the file management component 
82 may alternatively operate to cause the computer within 
65 the image recorder device to off-load image and transaction 
data. The off-loading of data may be made to remote storage 
devices schematically indicated 104 associated with con- 
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nee ted terminal devices to which data may be sent through 
the network 44. Of course in alternative embodiments of the 
invention other approaches and techniques may be used. 

FIGS. 5 through 9 are schematic views which represent 
the operation of components comprising executable instruc- 
tions in exemplary embodiments of the system. These com- 
ponents are preferably software components which operate 
in connection with the record acquisition component 68 and 
the device switching control component 66. In FIG. 5 a logic 
flow associated with motion detection is shown. The inputs 
from the cameras or the other input devices are processed by 
a detection area definition component 106. The definition 
component contains data and instructions representative of 
one or more detection areas in the field of view of particular 
cameras that are to be analyzed and/or disregarded for 
purposes of detecting motion. 

In some systems motion may be occurring fairly fre- 
quently within a field of view of the camera, but such motion 
is not of interest and it is desirable to not capture image data 
in response to such motion. For example when a camera is 
located in the security area from which the serviceable 
components of the banking machine are accessed, motion 
may normally occur within a portion of the field of view of 
the camera while in other portions of the field of view 
• motion only occurs when the machine is being accessed. A 
camera located in an ATM vestibule may have a window 
within its field of view. Activity occurring outside the 
window may not be of interest and optimally should not 
result in image data being recorded. Motion detected 
through the window is disregarded responsive programmed 
instructions in the motion detection component which 
excludes from the analysis movement detected within the 
window portion of the field of view. 

A camera positioned in the interior of an ATM housing 
may detect motion even when the service door of the 
machine is not open. This may occur due to flashing LEDs 
or other indicators within the interior of the machine. The 
detection area definition component 106 may define detec- 
tion areas that exclude such sources of light or motion from 
the motion detection analysis. In certain systems vibration or 
other regular movement may cause certain fixed objects to 
appear to move relative to a camera's field of view. The 
detection area definition component may be used to exclude 
from the analysis images of known objects within an area of 
normal movement. The detection area definition component 
establishes those areas of the field of view of each camera in 
which changes in the image indicative of motion are to be 
analyzed and/or those areas in which changes indicative of 
motion are not to be analyzed. It should be understood that 
the definition component may in alternative embodiments 
apply to other sensing devices such as infrared sensors or 
other sensor types which have a field of view for sensing 
regions in which activities are to be disregarded. It should 
also be understood that the definition component 106 may 
also be set such that all regions in a field of view which make 
up an image are analyzed for purposes of motion detection. 

The detection area definition component may preferably 
be configured remotely by authorized users at user terminals 
connected to the network. This is preferably accomplished 
by inputs which divide portions of the field of view of each 
camera into one or more areas where detected motion is.of 
interest and not of interest. Such areas are preferably des- 
ignated graphically on the output screen of a user terminal 
and are readily changed by inputs from authorized users. 

The detection area definition component communicates 
with a motion detection component 108. The motion detec- 
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tion component includes instructions which operate to com- 
pare sequential images obtained from the camera inputs. In 
one exemplary embodiment this is done by comparing 
intensities or color of corresponding pixels in sequential 
s images. The sequential images are analyzed at periods fairly 
close in time. Changes in intensity or color of corresponding 
pixels of greater than a threshold amount are counted or 
otherwise mathematically analyzed. Changes above the 
selected threshold for at least a selected number of pixels in 
jo me entire image or selected detection area(s) of the image, 
indicate a substantial enough change such that motion is 
considered to have been detected. When motion is detected 
in an area of interest the motion detection component signals 
a device within control component 110 which operates the 
15 device switching controller 66 and the record acquisition 
component 68 to acquire image data from the camera at 
which motion has been detected. The system may also move 
into more permanent storage image data captured prior to 
the triggering event depending on its programming. 
20 It should be understood that the motion detection feature 
is only used to capture images from those cameras for which 
the system has been programmed to acquire image data 
based on motion detection. In the exemplary system shown, 
. this is generally in the secure areas within the machine or an 
25 exterior area adjacent the area where a servicer performs 
operations. If the system is not programmed to acquire 
image data based on motion detection from a particular 
camera, motion within the field of view of that camera will 
not result in the more permanent storage of image data. 
30 As previously discussed alternative forms of the invention 
may operate to capture images on a generally continuous 
periodic basis. Such images may be temporarily stored in a 
queue or other memory and erased after a period of time. 
Systems of the invention may be programmed such that 
35 motion detection may be determined based on comparisons 
of pixels which make up these digitized images. The detec- 
tion of motion may also cause the system to operate in 
accordance with programmed instructions to retain one or 
more images from the queue that preceded image in which 
40 motion was detected, and to store these prior images in' 
correlated relation with the images captured in response to 
the triggering event. This feature enables an operator to 
review the conditions in the field of view of the camera prior 
to the triggering event. Such information will often prove 
45 useful in determining conditions or activities which led up to 
the triggering event. 

The memory configuration of the described embodiment 
provides advantages in that the system is enabled to capture 
image and transaction data while delivering image and 
50 transaction data from storage. As a result unlike prior art 
systems, the capture of image data does not have to be 
suspended while images are recovered or downloaded from 
the system. Further, the configuration of the system enables 
capturing image data from a number of sources virtually 
55 simultaneously. This solves a problem associated with cer- 
tain prior systems which when configured to detect motion, 
operate to record only from a particular camera where 
motion has been detected. Other image data cannot be 
captured while image data is being captured from the camera 
60 where motion was detected. This presents opportunities for 
compromise of such systems by creating a diversion at a first 
camera and then carrying out improper activities within the 
field of view of another camera. The preferred form of the 
invention does not suffer from this deficiency as image data 
65 may be captured in a plurality of cameras virtually 
simultaneously, and triggering the capture of images based 
on detection of motion at one camera does not suspend 
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programmed instmctions may operate to send the messages 
to different addresses depending on the time that the event 
occurs. Such messages may include electronic mail mes- 
sages which have one or more of the images captured 
S included therewith. 

The timing/sequence logic component 118 works in con- 
nection with a device switching control component 120. The 
device switching component 120 is operative to work in 
conjunction with the device switching controller 66 and the 
j 0 record acquisition control 68 to acquire image data from the 
selected cameras through the frame grabber. The device 
switching control component 120 may also be programmed 
in other embodiments to take other actions such as to operate 
or interface with alarm systems, automatic locking systems 
j 5 or other types of devices. In addition as previously described 
the timing/sequence logic component may also operate to 
temporarily acquire images from various cameras or other 
image capture devices on a periodic basis. The programmed 
associated with the particular triggering event 



image capture from other cameras. The system can also be 
delivering image and transaction data to a remote location 
while concurrently capturing such data from a plurality of 

The motion detection feature may operate in connection 
with an analysis component 112. The analysis component 
112 may be used in various embodiments of the invention to 
determine various information of interest. This may include 
for example to measure how long it takes a particular 
servicer to perform particular service functions within a 
machine or within a service access area. Alternatively, the 
analysis component may be used to determine how long 
customers remain watching an output device on the banking 
machine before, during or after a transaction is completed. 
This may be used to provide information concerning the 
degree of interest that a particular customer or customers in 
general may have in a particular type of promotional pre- 
sentation that is made at the automated banking machine or 
other output device. Such information may be recorded in 

connection with the data store and later used for further M ma y include storing on a more permanent basis 
analysis. Such analysis may include in the case of the images captured prior to the triggering event. These images 
servicer, comparing performance of service providers or may then be stored in correlated relation in the data store 
determining the relative ease of servicing of various types of with the images related to the event. Such information 
machines or components. It can also be used to determine if, enables an analysis to be made as the causes or events 
or for how long, a servicer had activity related to a compo- 25 preceding the triggering event. 

nent in the machine . In the case of customers and users, the FIG. 7 is a schematic view of the operation of the system 
analysis data may be used for targeting promotional type to acquire image and transaction data in response to soft 
information to users in the future as well as for evaluating trigger inputs. Such soft trigger inputs may include for 
the effectiveness of marketing type activities presented example messages to or from transaction function devices 
through the automated banking machine. The functions 30 on the interface bus within an automated banking machine, 
performed by the analysis component 112 on the captured Alternatively such soft trigger inputs may include transac- 
data will depend on the particular nature of the data to be lion messages transmitted between an automated banking 
analyzed^ but such analysis may be facilitated by the avail- machine and a host. Other types of soft trigger inputs may 

include receipt of other electronic messages either alone or 
in relation to other messages, so as to indicate a condition 
which requires image or transaction data acquisition. Other 
types of soft trigger events may be initiated in response to 
timing functions which operate based on programmed 
instructions and the current time, or which are timed from 



ability of image and transaction data which is stored 
correlated relation in the data store with the movement 
analysis data so that the validity of any conclusions made 
can be verified. 

FIG. 6 schematically represents a further aspect of the 
operation of embodiments of the present invention. FIG. 6 



represents an example of how the system operates to capture 40 other events. 



The soft trigger logic component 122 is operative to 
receive the soft trigger inputs and to analyze the nature of the 
conditions represented by the inputs received. For example 
the soft trigger logic component may determine based oh 
software instructions stored in memory that particular sig- 
nals on a bus or line being monitored represent the input of 
a customer card to a card reader and the account number 
associated with that card. In certain embodiments of the 
such account data is captured as part of the 



image and transaction data in response to hard trigger inputs. 
Such hard trigger inputs generally correspond to sensors 
which sense conditions or other activities adjacent to the 
machine. As schematically represented in FIG. 6, a sensor 
114 provides in input signal which is received by a hard 
trigger logic component 116. The hard trigger logic com- 
ponent is operative to determine the nature of the input and 
to communicate with a timing/sequence logic component 
118 which controls what occurs in response to the particular 

•input corresponding to a triggering event. 50 transaction record data and the input of such a'card to the 

For example the sensor 114 may be representative of a card reader is used as a trigger to capture image data so that 
sensor which senses when a service door on an automated there is a record of the user that input the card. Likewise 
banking machine is opened. The executable instructions messages indicative of the presentation of cash to a customer 
programmed in connection with the system include instruc- by a presenter may be detected and used as a further 
tions which comprise a sequence which controls what is to 55 triggering' event to capture image data, 
happen when this event is sensed. The timing/sequence logic In certain exemplary forms of the invention a series or set 
component 118 will generally include information that may of images is captured in connection with a transaction 
be time dependent, and/or a sequence of actions which are carried out by a user in an automated banking machine. Such 
to occur. The sequence may include for example having images in the set are preferably captured in response to the 
image data captured generally continuously from particular 60 operation transaction function devices on the machine Such 
designated cameras while the door is open. The sequence images are stored and may be recovered and displayed 
may further include sendmg one or more e-mail messages to together for later analysis. The storage of multiple images in 



particular e-mail addresses through the network so that 
individuals are notified that the machine has been accessed. 
As different entities may have responsibility for servicing 
machines depending on the date of the week or time of day, 
the routing of such messages may be time dependent and the 



a set related to customer transactions increases the likeli- 
hood that suitable images of the user and/or background will 
s be acquired which may prove useful later if such images 
require analysis. In addition, the fact that account data 
and/or other transaction data is captured in connection with 
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the image data and can be correlated therewith, enables 
searching the transaction data to recover the image data 
associated therewith. For example, because the transaction 
data commonly captured may include the account number as 
well as the user name encoded on the card, the transaction 
data may be searched using these parameters. This enables 
readily identifying transactions corresponding to these 
parameters and retrieval of the image data associated there- 
with. This greatly reduces the time to locate pertinent images 
compared to other systems. In addition, other types of 
sorting parameters may be used to recover images. These 
include for example, time periods during which transactions 
were conducted, amounts of deposits, amounts of withdraw- 
als or other transaction parameters. Any of these transaction 
parameters that are stored in connection with or which may 
be correlated to image data may be used to selectively 
identify and recover images. Of course in other embodi- 
ments other approaches to the capture of image data, trans- 
action data and other types of soft trigger logic may be used. 

Soft trigger logic component 122 operates in connection 
with a timing/sequence logic component 124. The timing/ 
sequence logic component is operative responsive to pro- 
grammed instructions input by a user during setup of the 
system. The timing/sequence logic component operates to 
capture image and transaction data selectively from various 
cameras and/or transaction function devices depending on 
events that are occurring and/or the date and time of such 
events. For example if particular transactions are occurring 
the timing/sequence logic component may take special 
actions different or in addition to those taken with regard to 
other actions. An example may be when a customer seeks to 
deposit more than a certain amount of funds in the machine 
or seeks to cash or obtain value for an instrument. The 
timing/sequence logic component may capture more fre- 
quent images or images from additional cameras during the 
transaction. Another example may be in the case of a 
reportedly stolen card. If the soft trigger logic identifies the 
input card as stolen, the logic component may operate to not 
only acquire additional image data, but also to send mes- 
sages through the system or through other communications 
channels to police or other authorities. 

A further example may be used in connection with a 
banking machine which includes check accepting or other 
document accepting devices where the authenticity of the 
inserted document may require verification. The timing/ 
sequence component may work in connection with an imag- 
ing device within the automated banking machine to capture 
an image of indicia on the inserted document, and to 
transmit an image of the document while the transaction is 
ongoing to a verification terminal in the network. Such a 
document may be viewed al such a terminal and/or elec- 
tronically analyzed to compare the image of the document to 
verification information such as a handwriting or signature 
database for purposes of determining authenticity. The des- 
tination where such messages are sent may be varied 
depending on the nature and/or amount of the document, the 
time of day and other parameters depending on the instruc- 
tions associated with the timing/sequence logic component 

Other exemplary applications of timing/logic sequence 
include minimizing the use of available image data storage 
by reducing or eliminating the amount of image data 
acquired related to certain transactions. For example the 
timing/logic sequence may include instructions to capture 
fewer or no image data related to transactions conducted that 
are of certain types. This may be appropriate for example in 
the case of an account balance inquiry. Likewise the ins true- 
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lions may provide that a dispense of cash below a particular 
amount, such as for example $100, may not result in the 
acquisition of image data. Likewise, certain deposit trans- 
actions for certain customers within certain limits may not 
S require the capture of image data, or may have the system 
capture a lesser number of images than is captured in 
connection with other transactions. 

The timing/sequence logic component 124 may operate in 
connection with instructions that capture additional image 
1° data in connection with certain transactions by certain 
individuals. Additional image or transaction data may be 
captured based on selected time of day, or a combination of 
time and day, amount or the nature of the individual cus- 
tomer. Various schemes for using customer profile data, time 
15 of day data and other information accessible through the 
network may be used in combination with the soft trigger 
inputs to selectively control the image and transaction data 
capture capabilities, and the message sending and device 
control capabilities of the system in response to selected 
20 circumstances that may arise in the operation of the auto- 
mated banking machine. 

A device switching control component 126 operates 
responsive to the timing/sequence logic component to cap- 
tare image data during the transaction. The device switching 
control component further operates to capture transaction 
data in connection with the transaction. This may include for 
example time and date data, account number data, amount 
data, transaction number data, user name data, machine 
3o location data and other data which can be derived from the 
soft trigger inputs or other information available to the 
machine. Such data may also include multiple items of 
similar data such as time data. This may be desirable for 
example when the ATM has an internal clock and the image 
3s storing device has its own associated system clock which 
may not be perfectly synchronized with the ATM clock. 
Capturing time data corresponding to both clocks may avoid 
confusion. Alternatively, programming may be provided for 
automatic clock synchronization and/or for obtaining time 
^ data or setting signals from another source. 

In exemplary embodiments of the invention, the nature of 
the related data analysis can be set by the user during setup 
of the system. This is done through a user terminal and is 
preferably accomplished by selecting options in a setup 
45 window such as shown in FIG. 25. The related data analysis 
and storage component 128 operates to capture and store the 
selected data. The data analysis and storage component is 
further operative to store the related transaction and other 
data in correlated relation with the image data. In certain 
J0 embodiments of the invention such correlation is provided 
by storing data representative of the time and date associated 
with the image data and transaction data. In other embodi- 
ments other approaches to correlate the image and transac- 
tion data may be used. 
55 In alternative forms of the invention the data storage and 
analysis component 128 may also include instructions for 
analysis of received data such as to provide statistical 
analysis related to use of the machine. Such data may be 
used in connection with developing a historical use pattern 
60 for the machine which may be used in connection with the 
memory allocation activities performed by embodiments of 
the system as later discussed herein. 

FIG. 8 is a schematic view of the logic flow associated 
with operation of embodiments of the invention where a lack 
65 of usable video information is detected with a camera that is 
to be operated in the course of a transaction. It should be 
understood that the lack of a usable video logic may operate 
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n with the motion detection logic, hard trigger 
logic or soft trigger logic previously described. 

Alack of usable video detection component 130 operates 
in response to executable instructions to determine if a 
camera that is or may need to be operated is not providing 
suitable image data. This is done in preferred embodiments 
by comparing pixel data from the areas of the image that are 
indicated to be of interest by the detection area definition 
component 106 or from the entire field of view. The lack of 
usable video component 130 determines if pixels which 1 
comprise an image are generally all above or all below 
certain intensity or color levels and/or are lacking in contrast 
across the image so as to not provide a suitable image. The 
logic may check for example if generally all pixels are 
indicated as dark, which may suggest that a camera is being j 
blocked or a lens has been spray painted so as to obscure the 
camera. Likewise the logic may check to determine if the 
pixels are generally all above a certain intensity value which 
may indicate that a glare condition created by reflected 
sunlight or a light operated by a person is obscuring a 2 
camera. The lack of usable video components may also 
operate based on detecting a rapid, large change in the field 
of view, or such a large change followed by an extended 
period without any change. Alack of usable video may also 
be based on detection of certain relatively unchanging high 
contrast images or sensing an unchanging image in a 
selected portion of a field of view. The lack of usable video 
component 130 may also be operative to detect that the 
camera signals have been interrupted. Various approaches 
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also interface with other devices and systems to carry out 
functions determined by the timing sequence logic. 

FIG. 9 shows an alternative logic flow used in connection 
with embodiments of the invention in which features of a 
user are used to identify and/or authenticate the user or 
actions carried out thereby. The logic flow represented in 
FIG. 9 includes an identification data acquisition component 
schematically represented 136. The identification data 
acquisition component in an exemplary embodiment oper- 
ates to acquire data with a camera concerning a physical 
feature of the user. For example camera 24 may be used to 
acquire camera signals corresponding to a face of the user. 
An identification processing component 138 is used to 
compare the image data acquired to image data correspond- 
ing to a set of authorized users. Such authorized user data 
may be stored in a data store. As schematically indicated this 
data store may be within the automated banking machine or 
may be accessible through a network. Such identification 
processing may process not only user image data but also 
other data such as data from an object provided by a user, 
voice data, iris scan data, retina scan data or other data that 
can be used to indicate that a transaction is authorized. 

If the identification processing component 138 is unable 
to identify the user then such information is provided to a 
machine control interface component 140. The interface 
component prevents operation of the machine but operates 
the system to capture image data related to the person who 
was unable to operate the machine. Alternatively if the user 
is identified as an authorized user by component 138, the 
machine control interface may authorize further operation of 



may be taken to making a determination that there is a lack 30 the machine, or may authorize such further operation if other 
" r """"" indicia such as voice, numeric or other inputs correspond to 

the authorized user. Again the machine control interface 
component will operate to acquire image data concerning 
the authorized user. A data analysis storage component 142 
operates to store data related to the transactions conducted 
by the authorized user and is operative to store transaction 
data in the data store. This may include the various types of 
transactions conducted by the user and may further include 
storing in correlated relation with the user data, data repre- 
sentative of instruments deposited by such a user, instru- 
ments produced for such an authorized user or other infor- 
mation related to the user's transaction which is stored for 
later recovery. The nature of the transaction information 
captured will depend on the nature of the automated banking 
machine and the image and transaction data captured in 
connection therewith. 

The capture of images from the various cameras on a 
continuing basis in embodiments of the system may also be 
used for other purposes. For example, the facial features of 
criminals, missing persons or other individuals of interest 



of usable video. 

A timing sequence logic component 132 operates respon- 
sive to component 130 to take action in response to the 
condition. The action is taken in accordance with a pro- 
grammed sequence which is set up by a user and stored in 
a data store. The sequence may include for example 
responding to a lack of usable video by capturing image data 
from additional cameras. For example if in FIG. 1 camera 24 
is unable to provide usable video, image data may be 
captured from camera 26. The programming of the system 
may also operate in response to detecting a lack of usable 
video event to store in connection with the event one or more 
prior images that had been obtained and stored temporarily 
from the camera which is considered to be no longer 
providing usable video. Such images may be useful in 
determining the cause of the loss of usable video and/or the 
identities of persons which may have caused the loss of 

e exemplary embodiments the timing sequence 



logic component in response to the lack of usable video may so may be stored in 
cause the server component to generate a message to 
selected addresses in the network to indicate the nature of 
the condition. Such messages may include therewith one or 
more images. Likewise the timing sequence component may ,„ w „ ui f 

formulate messages to service entities responsible for repair- 55 stored in the data 
ing the system to indicate that there is a problem. In 
alternative forms of the invention the timing sequence 
component 132 may operate to perform activities through 
additional interfaces or computers such as turning on alarms, 



with the data store. The system 



may operate so that content of images captured on a 0 
tinning basis from cameras, or alternatively images captured 
in response to triggering events, are analyzed so that the 
facial features of persons in images are compared to images 
stored in the data store. Responsive to finding a match the 
system may operate in response to programmed instructions 
to trigger a sequence which may include capturing addi- 
tional images, sounding alarms or sending messages elcc- 
.. . - ironically to selected individuals. Such messages may 

actuating additional hghtmg, contacting police authorities 60 include therewith the captured images as weU as information 



and/or disabling the automated banking machine. Such 
activities may be performed depending on the setup of the 
system as programmed by user. 

The timing sequence logic component 132 operates in 
connection with a device switching control component 134. 65 & Hauspie. Of course in other embodiments, other compo- 
The device switching control component operates to capture nents and approaches to recognizing persons and images 
image data responsive to programmed instructions and may may be used. 



concerning the person who was indicated to be recognized. 
Such facial recognition may be carried out for example in 
embodiments of the invention using software such as Face- 
software which is commercially available from Lernout 
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In addition, because embodiments of the invention 
include image data stored in response to transactions and 
other triggering events, the stored data may be retrieved 
using the parameter of facial features or a particular indi- 
vidual's appearance. This may be done for example to 5 
identify instances where a particular service person has 
worked on a particular machine. Alternatively transaction 
data may be reviewed to determine instances where a 
particular individual may have used the debit or credit cards 
of another person in conducting transactions. Numerous 10 
uses of searching through the image data using such param- 
eters may be used. 

Alternatively or in addition, the image data received by 
the system may be analyzed on a real time or periodic basis 
for the presence of other features in images. For example, 15 
images captured from a camera adjacent to an automated 
banking machine may be analyzed for the presence of 
certain objects which appear in the field of view of the 
camera. Such objects may include for example certain types 
of criminal tools used to attack the automated banking 20 
machine. Alternatively, objects which may be recognized 
may include certain types of weapons or other objects. 
Various body positions such as a person raising their arms or 
lying down might also be recognized. In response to a 
captured image having the image condition of including an 25 
object or characteristic which corresponds to one which is 
recognized by the system responsive to stored logic, appro- 
priate responsive actions may be taken. Again, such actions 
may include sounding alarms, shutting down the automated 
banking machine and/or sending messages including mes- 30 
sages which include images to programmed addresses or 
devices. Embodiments of the invention may operate in 
conjunction with or as part of a system as described in 
allowed U.S. patent application Ser. No. 08/813,511 which 
is owned by the assignee of the present invention and the 35 
disclosure of which is incorporated herein by reference as if 
fully rewritten herein. The identification of particular objects 
or features in the field of view of a camera may be operative 
to cause the dispatch of messages through one or more types 
of message media to predetermined recipients of such infor- 40 
mation. The dispatch of messages may include synthetic 
voice messages dispatched by phone or similar media, 
paging, radio messages or other types of messages. In 
addition, the responses to such messages may be monitored 
and tracked in accordance with programmed parameters to 45 
assure an appropriate response occurs. 

A further advantage of embodiments of the present inven- 
tion is that the stored image data is capable of being searched 
for other visual conditions or appearance features. For 
example, stored image data may be searched to uncover so 
images which were stored of users with certain facial 
characteristics. Such characteristics may include features 
that may be recalled by another person of a potential witness 
to an activity which occurred in the area where the image 
capture system is operating. Such image capture capability ss 
enables images to be sorted to look for persons with features 
such as certain hair color, facial hair, skin color, tattoos, 
earrings, jewelry, or glasses as well as for certain types or 
colors of apparel. This may include for example hats, ski 
masks, bandanas, ties and jackets. The ability of the present 60 
invention to sort through image data and to recover images 
based on one parameter or a combination of parameters 
enables the recovery of images that using prior systems 
would require considerably greater time and effort. As can 
be appreciated from the foregoing description, embodiments 65 
of the invention provide many uses and advantages com- 
pared to prior art systems. 
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FIG. 10 is a schematic view of an alternative form of the 
transaction record system of the present invention generally 
indicated 144. System 144 includes an automated banking 
machine which in the exemplary system is an automated 
teller machine schematically designated 146. Automated 
teller machine 146 is similar to the ATM described in the 
previous embodiments in terms of its outward appearance 
and configuration. However the computer and software 
architecture of ATM 146 differs. 

ATM 146 includes a plurality of transaction function 
devices 148. The transaction function devices include 
devices which can be used to carry out transaction functions 
with the machine. These are similar to the transaction 
function devices of the previous embodiment. The transac- 
tion function devices generally include input devices such as 
a card reader, keypad, touch screen and/or function keys. 
The transaction function devices may also include devices 
for dispensing sheets and currency such as a bill dispenser 
and bill presenter. The transaction function devices may also 
include a depository, printing devices for printing transac- 
tion receipts, printing transaction records and other docu- 
ments. The transaction function devices may also include a 
number of other devices. 

The transaction function devices are operative in response 
to a device manager/interface component 150. The device 
manager interface component may be comprised of applets, 
programs or other applications written in a language such as 
JAVA by Sun Microsystems or Active X by Microsoft. 
Component 150 preferably includes data and instructions 
which represent operational relationships among the 
devices, and such data and instructions are schematically 
represented by a data store in connection with component 
150. . 

The device manager/interface component 150 preferably 
operates the devices in response to HTTP format messages 
which are delivered by a device server 152. The device 
server 152 similarly includes a plurality of applets or other 
programs which operate responsive to messages received by 
the device server. The device server contains the instructions 
which generally operate to control, coordinate and limit the 
operation of the transaction function devices within the 
ATM. 

ATM 146 further includes a document handling portion 
154. Document handling portion 154 is operative to process 
HTML documents and HTTP messages which the document 
handling portion selectively accesses. The document han- 
dling portion 154 includes a browser for selectively pro- 
cessing HTML documents or other documents. The docu- 
ments accessed by the browser may include therein 
instructions such as JAVA script which are processed by the 
browser and which are operative to cause a computer to 
output messages through an output device such as a screen . 
display of the ATM. The document handling portion 154 of 
this example further includes a server device that is opera- 
tive to output messages to the other components of the 
machine as well as to a network 156 to which the machine 
is connected. The document handling portion 154 may 
access HTML or other documents through a bank server 158 
or other servers which are connected to the network 156. The 
bank server 158 may also send and receive messages from 
the device server 152 and other components of the machine. 
As shown schematically, the bank server 158 is in operative 
connection with a back office processing system 160. The 
back office processing system is operative to maintain data 
records and account information, as well as to provide 
information for generating documents and messages which 
are delivered by the bank server 158. 
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It should be understood that ATM 146 may be operated 
through messages exchanged with plurality of servers which 
are connected to the network 156. This may include other 
bank servers directly connected to the network 156 as well 
as bank servers which are connected to a further network 
162 which can be transmitted through a control server 164. 
An example of such a system would be a system in which 
network 162 is a wide area network such as the Internet and 
control server 164 serves as a firewall limiting the servers 
from which the automated teller machine 146 may receive 
instructions. 

It should further be understood that the document han- 
dling portion 154, device server 152 and device manager/ 
interface component 150 may in embodiments of the inven- 
tion comprise components which communicate through the 
operating system of the computer on which the components 
reside, or may communicate on a local area network which 
operatively connects the components of the machine. It 
should further be understood that in other forms of the 
invention the machine may be connected directly to the wide 
area network. 

In the exemplary embodiment of the invention shown in 
FIG. 10, the server component associated with an image 
recorder device resides on the computer which operates at 
least some of the transaction function devices of automated 
teller machine 146. An image server component 166 is 
resident on the computer within the automated teller 
machine and is accessible through the network 156 at an 
address on the machine. As in the prior embodiment, the 
image server is in operative connection with at least one data 
store 168. The data store 168 includes executable instruc- 
tions carried out by the image server as well as image and 
transaction data. It should be understood that the data store 
168 may represent a portion of overall memory available in 
connection with the computer operating the automated teller 
machine 146. Alternatively data store 168 may include a 
separate data store such as a recorder with a removable 
storage media or a combination of allocated storage avail- 
able on the computer in the machine and a separate data 
storage device. 

It should be understood that in certain embodiments of the 
invention the computer in the automated teller machine 146 
operates in a Microsoft Windows NT® software environ- 
ment and data storage is allocated between the components 
operating in the machine. Further the transaction data stor- 
age associated with the captured images accessible through 
the image server is shared with other transaction data storage 
maintained for transactions carried out by the machine, to 
reduce duplicate storage of data. Such transaction data 
storage information may be stored in the machine for 
purposes of archiving or accumulating batch data which may 
be later transferred to the back office 160 through the bank 
server 158 or to other locations. It should further be under- 
stood that in embodiments of the invention, image data may 
be downloaded to other devices connected to the network 
156 and accessed therefrom while transaction data may be 
maintained in storage at the ATM or in a different data store 
within the network. The downloaded data may be erased or 
overwritten after downloading to provide added storage 
space at the machine. Alternatively image data may.be 
downloaded with or at generally the time of each transaction 
at the machine. 

The exemplary form of the invention enables accessing 
image and transaction information from different locations. 
This is accomplished by coordinating image data and trans- 
action data which may be accomplished in embodiments of 
the invention by including with the image data, data repre- 
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sentative of source as well as information corresponding to 
a time associated with the transaction as previously 
described. This enables correlating the image data with the 
source transaction data corresponding thereto based on time 
5 and date. Of course other alternative approaches to recov- 
ering and correlating transaction and image data may be 

As shown in FIG. 10 image server 166 is connected to a 
hardware interface schematically represented 170. Hard- 

10 ware interface 170 is shown connected to cameras 172 as in 
the previous embodiment. Hardware interface 170 of the 
exemplary embodiment performs the switching, acquisition 
control, digitizing and hard trigger' receiving functions 
described in connection with the previous embodiment. 

15 Interface 170 may also be used to provide outputs for 
controlling camera aiming devices (such as pan/tilt/zoom), 
focus devices, lighting and other devices. It should be 
understood however that the allocation of such functions 
between a plurality of hardware and software components 
may be achieved in various ways within embodiments of the 

In the embodiment shown in FIG. 10 the image server 166 
is in operative connection with components 150, 152 and 
154 which are primary operational components of the ATM. 

25 Such configuration readily enables configuring the image 
server to cause the capture of image and/or transaction data 
in response to soft triggers which are in the form of events 
which are fired to components in connection with the server. 
Such programming may be readily accomplished through 

30 visual programming tools used in connection with program- 
ming in JAVA and other languages. Such programming tools 
may include Visual Age® by IBM and Visual Studio™ by 
Microsoft. Use of such programming enables readily estab- 
lishing and changing the soft triggers for image and other 

35 data acquisition as well as readily changing actions which 
may be taken in response thereto. 

As shown in FIG. 10 other terminal devices may be 
connected to the network 156. This may include user ter- 
minals 174 of the type previously described as well as 

40 verification terminals, data storage terminals and other types 
of terminals that work in connection with the system. 
Network 156 may be connected to interface devices sche- 
matically represented 176, which provide gateways to other 
communications mediums of the type previously described. 

45 Such gateways may be used for sending messages to 
servicers, police authorities or other persons who are to 
receive messages in response to events which occur at the 
ATM based on the sequence of configuration data for the 
capture of image data stored in connection with the image 

so server or other computer. 

As can be appreciated from the configuration in FIG. 10 
an authorized user operating a user terminal can access 
image data by accessing the image server with a browser and 
recovering image data from memory. This configuration 

55 further facilitates analysis of image data by being able to 
correlate transaction activity and the operation of transaction 
function devices with image data. Further the capability of 
the exemplary embodiment of the invention to capture 
image and transaction data while virtually simultaneously 

60 delivering image and transaction data to a remote user, 
facilitates maintaining ATM 146 in operation. Actions in 
response to triggering events may include panning, tilting or 
zooming cameras which may be used to verify suspect lack 
of usable video events or as actions in a sequence. Other 

65 advantages of this embodiment due to the flexibility and the 
ability to readily make changes in configuration will be 
appreciated by those skilled in the art. 
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An alternative form of the system of the invention gen- 
erally indicated 178 is shown in FIG. 11. The system 178 
includes an automated banking machine which is an auto- 
mated teller machine generally indicated 180. ATM 180 is 
similar to ATM 146 previously described except as dis- 
cussed herein. 

ATM 180 includes a computer which includes an image 
server 182. Image server 182 operates in a manner similar to 
image server 166. However image server 182 instead of 
acquiring image signals through a hardware device obtains 
image signals from a connected network 184. In the system 
shown in FIG. 11 cameras 186, 188 and 190 are each 
connected to a mini server 192, 194 and 196 respectively. 
The cameras and mini servers are each operative to function 
as a network node in connection with network 184. Each 
network node includes hardware and software which con- 
verts the camera signals to image pages or similar image 
files that can be transmitted through the network 184. These 
images can be relatively spaced in time or close enough 
together to be considered as full motion. The programmable 
instructions executed in connection with image server 182 
arc operative to selectively access the cameras through the 
associated mini server and to download images therefrom. 
Such images may be stored as image data in correlated 
relation with transaction data in the data store within the 
automated teller machine. Alternatively image data may be 
stored in data stores associated with each of the mini servers 
so that it may be selectively accessed therefrom by image 
server 182 as well as from other authorized terminals within 
the network. 

As can be appreciated, this alternative configuration fur- 
ther distributes the acquisition of image data and transaction 
data. However as the transaction data is accessible through 
the image server 182, and the system location of the mini 
servers 192, 194 and 196 are each known from their asso- 
ciated URL or similar system address, correlation and recov- 
ery of image and transaction data may be readily accom- 
plished. It should further be understood that while in the 
configuration of the system shown in FIG. 11 each camera 
is shown with an associated mini server, a group of several 
cameras may be interconnected and may selectively deliver 
image data through a single mini server to the network. 
Alternative configurations may be used to suit the particular 
nature of the system being operated. 

FIG. 12 shows yet another alternative system of the 
present invention generally indicated 198. System 198 
includes an automated banking machine which is indicated 
as ATM 200 which may be generally similar to ATM 146. 
ATM 200 is connected to a network 202. A computer 
including an image server 204 generally similar to image 
server, 166, operates on ATM 200. Cameras 206, 208, 210 
and 212 operate to supply camera signals which are received 
by image server 204 through an interface 214. In this 
embodiment the interface 214 is an interface to a second 
network schematically indicated 216 in which the cameras 
are connected. The interface 214 in one form of the inven- 
tion is an interface to a power supply network to which 
cameras are connected. Interface 214 may be for example an 
interface to a power distribution system within a facility in 
which the ATM is operated. An X-10 technology type of 
communication may be used for example. Signals from the 
cameras 206, 208, 210 and 212 are superimposed on the 
power distribution line through a plurality of impedance 
matching interfaces 220, 222 and 224 respectively. Signals 
sent by interface 214 are operative to cause selected ones of 
the cameras to output camera signals superimposed on the 
power distribution lines. Such image signals may be 
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received at interface 214 and processed in the manner 
similar to other camera signals as previously described. 
Camera signals sent in the second network may take various 
forms of analog and digital signals and may be multiplexed 

5 or otherwise sent simultaneously so that image data may be 
acquired and captured selectively by each of the cameras as 
described in connection with the previous embodiments. 
Signals for controlling or positioning cameras may also be 
transmitted through the network as well as image data. 

jo FIG. 13 shows yet another form of the invention referred 
to as system 226. System 226 includes an automated bank- 
ing machine 228. Machine 228 is an ATM similar to ATM 
146 except that it includes among its transaction function 
devices a check or other document imager schematically 

15 indicated 230. ATM 228 operates to accept checks or other 
instruments from users of the machine in response to control 
by the other components. The imaging device 230 operates 
to produce document image signals representative of docu- 
ments that may be deposited or received by a user in the 

20 machine. An image server 232 or a computer in which it 
operates is operative to cause the capture of images pro- 
duced by the imaging device and store image data respon- 
sive thereto in the associated data store. In addition, the 
computer is operative to cause the machine to capture 

2j transaction data and/or to correlate transaction data captured 
by other components of the machine, with image data. 
Image server 232 and the associated computer may also 
operate in connection with cameras and other input devices 
similar to those discussed in connection with the previously 

30 described embodiments. The computer may further store 
camera image data in memory in correlated relation with 
document image data generated from the imaging device. 

Image server 232 is in operative connection with a 
network 234. Network 234 is in operative connection with 

35 a terminal 236. Terminal 236 may serve as a document 
verification terminal. Terminal 236 has in connection there- 
with a verification data store schematically indicated 238. 
Verification store 238 includes therein data representative of 
indicia which can be used to verify genuineness of docu- 

40 ments input to the machine through the imaging device. For 
example verification data store 238 may include data rep- 
resentative of customer signatures and/or other identifying 
data for customers authorized to provide checks into the 
machine. 

45 Document verification terminal 236 includes a computer 
including a browser therein. The terminal 236 is controlled 
responsive to input devices that access document image data 
through the image server 232. The document verification 
terminal 236 operates responsive to the document image 

50 data to compare indicia in or associated with the document 
image data, to indicia stored in the verification data store. 
This may be done for example by comparing image data 
related to checks or similar documents input to the check 
imager 230 to images of known genuine signatures stored in 

55 data store 238. Such indicia may be compared for genuine- 
ness by human comparison on a side-by-side basis by 
outputting such information to an output device such as a 
screen. Alternatively the data may be manipulated to place 
such signature data in overlapping relationship or in other 

60 relative positions so as to facilitate analysis thereof. 
Alternatively, verification terminal 236 may include instruc- 
tions such as software programs which are operative to 
compare indicia in document image data to indicia stored in 
data store 238. Such verification software may compare the 

65 signature data from the input document and the known 
genuine signature and provide an indication of suspect 
signatures or possible forgeries. This may be accomplished 
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by comparing the image data corresponding to contours of 
letters, portions of letters or combinations of letters within a 
signature, and indicating. when a level of correspondence 
does not exceed a particular threshold. 

Image server 232 may have associated instructions which 
causes document image data to be provided automatically 
periodically to verification terminals 236. Alternatively 
image server 232 may be configured to operate in connec- 
tion with other components of the machine to provide an 
indication during a transaction involving an instrument, and 
to forward such document image information through the 
network 234 so that the character or genuineness of the 
deposited document may be verified before the transaction is 
completed. This has the advantage in that when cameras are 
used in connection with the machine, an image of the 
individual operating the machine as well as the document 
image data may be viewed or processed before crediting or 
charging the customer's account for the value of the depos- 
ited or dispensed document respectively. The ability to 
capture the image of the customer along with the document 
image and to store the two in correlated relation further 
facilitates tracking and minimizes fraud. In addition, the 
verification terminal 236 may operate in the manner previ- 
ously described in connection with user identification soft- 
ware which enables identifying a user by physical charac- 
teristics. This further minimizes the risk of fraud. 

It should be further understood that although the exem- 
plary embodiment has been described in connection with a 
document imager and an attended verification terminal 236, 
embodiments of the invention may operate using unattended 
verification terminals such as terminal 240 which operates to 
carry out verification activities according to stored instruc- 
tions without human interactions. Alternatively other forms 
of the invention may verify the authenticity of deposited 
documents through watermarks, holograms, inks having 
magnetic, fluorescent or other characteristics or other indicia 
which is indicative of genuineness of deposited documents. - 
Other approaches and configurations may be used depending 
on the nature of the documents being accepted or dispensed 
and the indicia which must be compared or processed in 
order to determine the genuineness of the accepted docu- 

It should further be understood that features of the system 
shown in FIG. 13 may be applied to systems in which 
documents are printed with identifying indicia so as to 
enable more ready verification of their genuineness. This 
may include for example printing an image of a user on a 
check or other document dispensed by the machine. This 
may be done by the image server in response to image data 
from a camera which has the user in its field of view during 
the transaction. Such image data may be delivered by the 
image server to the printer which is one of the transaction 
function devices in the machine. The image data may be 
used by the printer to produce a document which includes 
the image of the authorized user. This reduces the risk of the 
document being presented by unauthorized persons. In addi- 
tion or in the alternative a computer in connection with the 
image server may obtain image data concerning an autho- 
rized user, watermarks or other information from memory or 
from terminals connected to the network 234 or may gen- 
erate one or more identifying numbers or other indicia, and 
include such information or indicia in printed documents it 
produces. 

Further alternative forms of the invention enable corre- 
lating image and transaction data for documents received or 
produced by the machine. This enables users at other ter- 
minals which have access to the network 234 to verify the 
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appearance features, such as the appearance of a person to 
whom a document was issued. This enables persons accept- 
ing such documents to verify the authority of the person 
presenting the document to possess it. In addition if the 

5 document is redeemed at another terminal, the image of the 
person redeeming the document may be compared to the 
image of the person who received the document to verify 
that the document is being redeemed appropriately. This 
may be done visually using an output device at the terminal 

1Q where the document is redeemed or may be done at a remote 
verification terminal in the network by an operator or by 
image comparison software. Alternatively identifying indi- 
cia in a presented document may be checked for genuineness 
and/or validity. For example, the redemption of documents 

15 may be recorded and tracked, so that upon presentment a 
check is made as to whether the presented document has 
already been redeemed. 

Similar principles may be applied with regard to data 
representative of value which is loaded onto smart cards or 

20 similar instruments. Data representative of the image of the 
person who has received the value may be stored in corre- 
lated relation with indicia corresponding to the transaction in 
which value is loaded and/or with identifying indicia asso- 
ciated with the card. Later when an individual presents that 

25 same card at the same or a different terminal, an image of the 
person presenting the card may be captured and/or the 
appearance of the person may be compared to the image data 
stored in memory. Image data of the authorized user may 
also be stored in memory on the smart card. Such image data 

30 may correspond to facial features. Alternatively image data 
may correspond to other features that are capable of being 
viewed by eye or read with the aid of a machine such as 
fingerprints and iris scans. Similar principles may be applied 
to other types of transaction systems and devices to mini- 

3J mize the risk for fraud and abuse. 

Forms of the present invention may enable the manage- 
ment of available memory to minimize the risk that image 
data and/or transaction data related to transactions con- 
ducted at the machine will not be captured and stored in 

40 memory. FIGS. 14 and 15 schematically represent steps 
performed by certain forms of the invention to manage the 
amount of memory resources and to selectively off-load 
image data when necessary. In addition the exemplary form 
of the logic described in connection with FIGS. 14 and 15 

45 is operative to estimate when memory resources such as a 
permanent image storage medium will become full based on 
transaction rates, and to forward a message to appropriate 
personnel of such impending loss of memory capability. 
Referring to FIG. 14, the logic flow commences with a 

50 step 242 in which a decision is made as to whether image 
data has been stored. If so, a determination of available 
memory is made in a step 244. In addition a record is made 
as of the available memory as of the time and date of the 
transaction. This is done at a step 246. The decision is then 

55 made at a step 248 as to whether the available memory is 
below a particular threshold. If so, certain actions are taken 
as described in connection with FIG. 15. 

If the available memory is not below the threshold as 
determined in step 248 a determination is made at a step 250 

60 to calculate memory use over the preceding set number of 
days, hours or other time period. At a step 252 the calcula- 
tion is then made as to a time to depletion (TTD) based on 
the current rate of memory use. The determination is then 
made at a step 254 as to whether the time to depletion (TTD) 

65 is less than a set number of days. If so, actions are taken 
similar to those taken when the available memory is below 
a threshold as described in connection with FIG. 15. 
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If the time to depletion is less than the set threshold, the 
logic flow then operates to recall from memory historical use 
pattern data. This is done at a step 256. This historical use 
pattern data may be information regarding the level of use of 
the memory based on the day of the week or other corre- s 
latable data for the machine over a period of time. Such 
pattern data may involve fuzzy logic or other programming 
which may make allowances for pay periods, holidays, 
vacation periods and other activities which are used to 
establish the historical model on which the pattern is based. JQ 
Using the historical pattern data the logic flow calculates an 
estimated time to depletion based on the pattern data in a 
step 258. The time to depletion based on the pattern data is 
then compared to the threshold in a step 260. If depletion is 
expected to occur based on the pattern data in less time than 
the set threshold, action is taken. If the time to depletion is 
longer than the set threshold the pattern data is updated in a 
step 262 and the logic flow is repeated the next time a 
transaction occurs. 

It should be understood that although in this described 2Q 
logic flow three determinations are made as to available 
memory, in other embodiments a lesser number of tests or 
additional tests may be made. In addition the tests may be 
correlated or combined using fixed or fuzzy logic type 
principles to calculate a time when depletion is expected. ^ 

In the event that there is concern about lack of memory as 
determined in steps 248, 254 or 260 a determination is made 
at a step 264 concerning whether the instructions associated 
with the image server include executing an image download 
sequence prior to the memory reaching capacity. If so an 30 
image download sequence is executed at a step 266. This 
image download sequence may be to a remote terminal 
through the network. Alternatively the download sequence 
may be to a hard or soft permanent or temporary storage 
device. Such download sequence also includes clearing the 35 
portion of the memory that becomes available after data is 
downloaded or otherwise allowing the memory to be over- 
written such that additional image data may be stored. 
Banking machine data which identifies the particular 
machine which generated the image and transaction data 40 
may be added to or stored in correlated relation with the 
downloaded data in accordance with programmed instruc- 
tions to facilitate analysis after the data is downloaded. 

If the computer and associated image server is not con- 
figured to conduct an image download, a determination is 45 
made at a step 268 concerning whether available memory 
may be reallocated. In some circumstances the memory 
allocated for storage of images may be expanded to include 
additional memory. This may include for example a dynamic 
reallocation of memory storage by the operating system of 50 
■ the machine based on resources being utilized. Such 
memory may be allocated on a temporary or permanent 
basis. If memory reallocation functionality is provided a 
reallocation sequence is executed in a step 270. 

If memory reallocation is not available a determination is 55 
made at a step 272 as to whether a notification message 
concerning impending depletion of the memory has been 
sent within a given time window. If a message has been sent 
within the time window then no further action is taken. 
However if a message has not been sent within a given time 60 
window a message is formulated by the image server at a 
step 274. This message preferably includes data as to the 
particular machine and when the available memory will 
reach depletion based on the current rate of transactions, 
historical data, threshold value or other basis upon which the 65 
determination to send the message was made. After the 
s formulated the device server executes the mes- 
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sage sequence and operates to send the message to the users 
who are to receive it based on the image server configuration 
and the instructions stored in the system. Generally such 
messages will be sent as one or more e-mail messages to 
selected e-mail addresses in the network. Of course in 
alternative embodiments other types of messages may be 

FIGS. 26 and 27 show examples of user screens which are 
presented by the image server to user terminals as part of a 
configuration sequence. Through use of the templates estab- 
lished through these setup screens users are enabled to 
configure individual e-mail and group e-mail lists. These 
lists include persons to be notified in the event that particular 
events occur. The notification of particular individuals at 
e-mail addresses is included as part of the timing and 
sequence instructions stored in connection with the image 
server which determine what is done in response to particu- 
lar events. 

As later discussed in detail alternative forms of the 
invention may operate to selectively delete stored image 
and/or transaction data. For example, transactions may be 
identified by selected parameters and image and/or transac- 
tion data associated with those transactions may be deleted. 
This may be done based on parameters such as elapsed time 
since the transaction was executed. Alternatively, transac- 
tion data may be deleted based on the type of transaction, 
amount or other triggering event associated with the image 
data. Thus, for example, data associated with withdrawal 
transactions which are under a certain amount and which 
occurred more than a particular number of days previously, 
may be deleted in response to programmed instructions. This 
frees up available space for storing data associated with 
additional transactions while to preserving image and/or 
transaction data related to other transactions which may be 
more significant. Similarly, image or transaction data cap- 
tured in response to other types of triggering events such as 
alarms, servicing activities or other conditions which cor- 
respond to a particular parameter or combination of param- 
eters may be stored for longer periods of time prior to 
deletion and/or downloading from a local memory. Various 
parameters for the preservation or deletion of data may be 
developed based on the nature of the system, the transactions 
conducted and the needs of the system operator. 

Alternative embodiments of the invention may operate to 
advise a person who is setting up sequences or operation of 
the system, about how long the system will be able to run 
before image data will need to be deleted or off loaded. The 
computer operating to store data or in connection therewith, 
may store historical use data for the ATM or other machine. 
Such historical use data, combined with the number of 
images that the system is configured to capture and the 
degrees of associated data compression (as well as possibly 
other data) may be used to calculate a period of time until the 
available memory is used. Alternatively, and particularly 
when no historical use data is present, the computer may be 
programmed to prompt a user to provide estimates of the 
number or frequency of triggering events and/or transaction 
rates. This information may be used by the computer to 
calculate how long the system can operate without deleting 
or off-loading images. The user in response to the output of 
such estimates may choose to change settings or sequences 
to capture more or fewer images in response to each trans- 
action or event, or to change the degrees of data compres- 
sion. In addition the computer may be configured to send a 
message to a selected user or address if transaction rates 
change from the historical or estimated rates by more than 
a set amount, and advise of the time period available based 
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on the actual rate of memory use. In response to such a 280 various sequences may be executed responsive to trig- 
message a user may choose to reconfigure the system. gering events such as detection of motion in fields of view 
The described form of the present invention presents a of various cameras, the blocking of one or more cameras (at 
useful user interface which may be used to set up the system ^ time or durin g a time of desired image capture), in 
configuration. Generally such configuration is established s response t0 various transaction functions carried out by 
from a user terminal which is connected to the image server Iransacuc.il function devices or on a periodic time schedule, 
through a network. In this example the image server con- Screen Z shown m FIG - 25 B ffl exemplary screen 
figuration provides for three levels of activities which users P res « nled »». a "»r terminal which enables a user to sr' - 



are authorized to perform. These levels correspond to cat- *L ™££n t^L* ^ ™ l ° 'f^'*'! 

f< goriesofpriv fl eg 6 s^ 

service . A screen 278 shown in FIG. 16 shows the cat- data can be selectively captured. This is done from 



r . * - , „ ... uaia wan acicvLivciy uapiuicu. mas is aone irom screen 

egones of activities and the user groups which are permitted 288 by selecting types of transaction data to be captured, 

to perform them in accordance with the configuration of an i ma g e data may also be captured in response to the operation 

exemplary embodiment of the invention. D f selected transaction function devices and responsive to 

As previously discussed, certain forms of the invention 35 the type of transaction function devices resident in the 

enable the configuration to include timing and sequence data machine. 

which specifics what images and data to capture, as well as In the event that the sequence configuration includes 

what further actions to take in response to certain triggering sending c-mail messages to selected addresses, the image 

events. FIG. 20 shows a screen 280 which may be displayed server is operative to send such messages in accordance with 

at a user terminal to establish a sequence of events that occur 20 e -mail information which has been stored in connection 

in connection with particular events. Such sequences may be therewith. Screen 290 shown in FIG. 26 is a template for a 

programmed so that the sequences are different based on the user to ^ ™ inputting e-mail address information for 

day of the week and/or the time of day individuals. Individual e-mail addresses may be combined 

T „ „. . ... . . c ■ ... , into e-mail groups and a screen 292 shown in FIG. 27 may 

In accordance with the user interface m this exemplary 2J be accessed t0 SQ0W the of individuals who are 

embodiment sequences are programmed by establishing a notified respons ive to events which may occur at the termi- 

daily schedule of what is to occur in response to events. FIG. nal . The configuration of ^ terminal Sllch that a phlrality 

21 shows a screen 282 which is presented in response to of individuals may be sent an e-mail message in response to 

clicking on the "daily program" icon from screen 280. the occurrence of a single event or other activity at the 

Screen 282 enables a user to configure the program to 3Q terminal. This facilitates the notification of individuals in 



establish what is to occur if particular events occur within a 



it that several individuals may be required to respond. 



given time window. In programming .of mis embodiment, ,f M previouslv the timmg aspect of 

multiple sequences overlap days, the narrowest schedule med ces enabks different tQ be ^ 

overrules broader schedules For example if a schedule .s fled of events at djfferent ^ ^ on d ^ 

configured for weekdays but a different schedule is config- J5 faciJitates notif ; (he s whf) have tQe m J t 

ured for a specific day the specific day schedule wtil responsibility for the at the time it occurs. Forms 

overrule the general schedule for that day. Likewise to of the ma ^ be conflgured to atlach or 

prevent madverten overlap of sequences, the programming m c . mails ; wbich co nd to the triggering evcnt 

of this embodiment provides entering only a start time for a which causes ^ „ otilication to be sent. This may immedi- 

3T e " ^ ^ 1S not ref3 T Jtf^T wdI 40 atel y P r0vidc me P CROn recciviD g the e " mail ^ 

contmue until a new sequence is begun. FIG. 22 shows a j,^,^ ^ what js occurrmg at the machlne . Aseries 
screen 284 used in an exemplary embodiment of the inven- of jm Qr kts fof , hc modifi( f ation of im m also 
lion. screen ^84 is generated responsive to selection ot the 5e transmitted with the notification. This may include for 
ry y ic n rom screen example images which occurred prior to the triggering 
Actions in a sequence are established by selecting the 45 event. Such e-mails may also include information about the 
"setup sequence" icons shown in screen 284. Selecting such nature of the triggering event, the location or banking 
an icon that is active generates a screen 286 of the type machine where such event is occurring and other pertinent 
shown in FIG. 23. Screen 286 enables a user to establish the data. In this way, the entities notified will receive a record of 
degree of data compression for images captured during the what has or is happening at the machine. This record will 
sequence. The compression level can be modified such that 50 also be available even if the machine is compromised and 
different sequences of events cause images to be captured at rendered inoperative shortly thereafter. Embodiments of the 
different compression levels which produce different image invention may also include with such image files, digital 
quality levels. Generally the less the data is compressed the watermarks or other indicia of authenticity so that the 
higher the image quality. However available memory is used accuracy of the information provided and the images asso- 
more quickly when the degree of data compression is less. 55 dated therewith have enhanced assurances that they have 
In this exemplary embodiment of the invention a plurality not been tampered with. Further, included in e-mails or 
of actions may be added to a sequence by clicking on icons attachments thereto may be sound or other files with which 
such as "add camera", "add output" or "add e-mail". In images are associated. This may be accomplished through 
alternative embodiments, additional actions may include the programming of sequences which include the capture of 
"repeat sequence" and "wait" type actions. Clicking on such 60 audio or other data in response to the occurrence of trigger- 
icons changes the system configuration so the system will ing events. Numerous alternative approaches may be taken 
take actions in a sequence such as those previously dis- utilizing the principles of the invention. Of course, embodi- 
cussed. Such sequences may include for example input of ments of the system may carry 



instructions for capturing images from cameras, sending other than through e-mail such as by RF, fax o 

e-mails to individuals or groups of individuals, providing 65 voice communication through telephone coonec 

selective outputs to the control devices, or sending messages As previously mentioned, security associated with the 

through the to network. As can be appreciated from screen image server may be important to prevent accessing by 
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unauthorized individuals. In the exemplary form of the 
invention password protection is provided to minimize the 
risk of unauthorized use. Of course in other embodiments 
other security techniques such as public key encryption, 
encryption of image and transaction data and digital signa- 
tures may also be utilized. FIG. 24 shows a screen 294 which 
is used in embodiments of the invention to establish access 
for particular users. A system administrator is enabled to 
gain access to screen 294 and to input information concern- 
ing additional users. Screen 294 also enables the system 
administrator to establish passwords to be used by each 
authorized user. 

Embodiments of the invention may also restrict certain 
users, or certain categories of users, in the type of image data 
that may be reviewed. This may be done in exemplary 
embodiments by limiting access to image and/or transaction 
data selectively to users, based on the types of triggering 
events associated with the storage of images. Alternatively, 
certain users may be precluded from viewing images cap- 
tured from certain cameras. This capability may be used to 
prevent certain users from observing certain images such as 
images which may include customer PINs or the combina- 
tion to a lock on an ATM. By preventing selected users from 
accessing certain image data based on the type of triggering 
event or camera associated therewith, images captured by 
the system that need not be restricted may be made available 
more broadly and used for potentially more purposes. 

A useful aspect of embodiments of the present invention 
is the ability of the system to provide screens or displays of 
image and transaction data that can be readily sorted, viewed 
and analyzed at user terminals within the network. FIG. 17 
discloses a screen or display 296. Display 296 includes sets 
of images 298, 300, 302 and 304. Each image setincludes 
"thumbnails" of five images. Each set corresponds to a 
transaction carried out by a particular user and each set of 
thumbnail images which comprises a set, corresponds to 
images of the particular user during that transaction. Of 
course it should be understood that in situations where the 
timing and sequence programming require a lesser or greater 
number of images, the number of images which comprise a 
set may differ. In addition as previously discussed, some 
transactions or triggering events may have no corresponding 
images at all. Other events which do not correspond to ATM 
transactions may have a large number of images spaced 
closely in time depending on the configuration of the system. 
This may include full motion or image frequencies 
approaching full motion. 

The images which have been captured and stored by the 
system may be preferably arranged in one or more series. A 
series may be a collection of all stored images arranged 
chronologically. Alternative series may be produced by 
segregating images that correspond to one or more types of 
triggering events or transaction parameters. Images included 
in such a series may be ordered chronologically, may be 
ordered in a hierarchy in accordance with one or more search 
parameters, or other ordering scheme. A useful aspect of 
some embodiments is that the user terminal enables a user to 
scroll through a series of images, displaying one or more of 
the images on the display at a time, by selecting certain icons 
with an input device. The icons enable the user to selectively 
display images and to move to display one or more different 
images at points forward or backward in the series from an 
image or images currently being displayed. In exemplary 
embodiments, selection of certain icons cause the display to 
change and display images in different increments and in 
different directions in the series from one or more images 
currently displayed. 
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In an exemplary embodiment screen 296 includes icons 
306, 308, 310, 312, 314 and 316. The icons may be used to 
selectively scroll through sets of images and images in the 
sets. As explained with reference to an exemplary help 

s screen 318 shown in FIG. 18, selecting icons 310 and 312 
enable scrolling backwards and forwards respectively by 
one event. Selecting icons 308 and 314 enable scrolling 
backwards and forwards respectively by an increment of ten 
events. Icons 306 and 316 enable scrolling backward and 

10 forward respectively to the beginning or end of a series of 
events or images. 

Exemplary screen 296 also includes "jump to image" and 
"jump to event" input boxes 320 and 322, respectively. As 
explained in FIG. 18 boxes 320 and 322 may be used to 

15 select images that are to be displayed. A "save comments" 
box 324 is used to selectively store comments in correlated 
relation with particular images. Comments can be manually 
input, input by voice as sound files, input through voice to 
text conversion software or may be generated and stored in 

20 response to programmed instructions based on parameters 
and/or triggering events. 

Screen 326 shown in FIG. 19 shows a selected image 327 
which has been enlarged by selecting one of the images from 
the sets. This may be done in the described embodiment by 

25 clicking on an image with a mouse or through other inputs. 
As shown in screen 326, the enlarged image 327 is displayed 
with corresponding transaction data which corresponds to 
the image. In addilioo event and image data corresponding 
to the image is also displayed. A user reviewing the image 

30 data is enabled to review any of the available image and 
transaction data. 

Advantages of the described embodiments of the present 
invention include the ability of a user terminal to access 

3S image and transaction data selectively. For example through 
operation of the browser or other programs within the user 
terminal an authorized user is enabled to search for selected 
parameters such as user name, account number, time and 
date and other data which may be stored in the data store. 

40 Image and transaction data may also be searched by com- 
binations of parameters or ranges of parameter values. This 
enables the operator of the user terminal to find selected 
image data rapidly or more selectively, and without having 
to scan through large volumes of information. In addition the 

4j exemplary forms of the present invention enable holding 
image and transaction data for substantially longer periods 
of time with minimum inconvenience. As a result this 
enables such data to be analyzed for much longer time 
periods and potentially much more inexpensively than is 

50 currently possible. 

A further advantage of preferred forms of the present 
invention is that image data is readily accessible and search- 
able. This facilitates identification in connection with issued 
documents such as bank checks or value loaded to smart 

55 cards as previously discussed. This enables users having 
access to the data to verify that a document or other item is 
being presented by an authorized user by accessing and 
visually or automatically comparing image data. Further 
advantages and novel aspects of the invention will be 

60 apparent to those having skill in the art. 

FIG. 28 shows yet another example of a system of the 
present invention designated 328. System 328 is similar to 
other systems of the invention previously described except 
as discussed herein. In system 328, image capture and 

65 delivery functions are performed by a separate device 330. 
Device 330 in this embodiment includes one or more 
computers or processors therein including one or more 
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servers, and is operative to capture and store image data, 
transaction data and other information from devices to 
which it is connected. Device 330 also includes appropriate 
interfaces to communicate with the devices to which it is 
connected for purposes of receiving inputs and outputs. As 
schematically indicated in FIG. 28, a computer included in 
device 330 is in operative connection with a data store for 
purposes of storing instructions as well as image and trans- 
action data. It should be understood that while a single 
device for performing the functions is shown in system 328, 
other forms of the invention may include a plurality of 
operatively connected devices including a plurality of pro- 
cessors and operatively connected data stores as well as 
other computers and interfaces, to perform the functions 
similar to that of device 330 described herein. 

In system 328, device 330 is connected to one or more 
automated banking machines schematically indicated 332. 
Automated banking machine 332 is similar to the machines 
previously discussed and includes a plurality of transaction 
function devices. Automated banking machine 332 may 
have one or more cameras or other image capture devices 
adjacent thereto as represented by camera 334. As will be 
appreciated, a number of cameras may be positioned adja- 
cent to the machine by being within and/or near to auto- 
mated banking machine 332 for purposes of capturing image 
data related to users, documents, surroundings or other types 
of visual inputs that may be desirable to capture and analyze. 
Camera 334 is operatively connected to device 330 such that 
device 330 may receive and capture image data therefrom. 
It should be understood that additional types of data capture 
devices may also be included adjacent to or within auto- 
mated banking machine 332. This may include for example 
microphones for capturing sound or voice information as 
well as devices which capture data related to transactions. 
Embodiments of the present invention can use voice recog- 
nition software to detect sounds from the microphone rep- 
resentative of words or the stress levels of sounds emanating 
from persons near the automatic banking machine. Such 
voice or sound data may be used in combination with images 
or other data to further detect and evaluate conditions at or 
near the automated banking machine. The data or informa- 
tion which is captured is also communicated to the device 30 
through one or more appropriate electronic connections 
schematically indicated 336. 

In addition to capturing images or other data from one or 
more automated banking machines, system 328 may also be 
operative to monitor one or more other transaction devices, 
as well as to monitor and record activities which occur 
within a facility. One or more cameras represented by 
cameras 338, 340 and 342 are shown and are representative 
of cameras used for this purpose. The cameras may be used 
for capturing images in response to triggering events, which 
may be either hard or soft triggers from one or more types 
of input devices. Alternatively, the cameras may capture 
images on an ongoing basis in one or more sequences for 
purposes of providing a generally continuous record of 
overall activity within an area. As in the previous 
embodiments, this form of the invention also provides the 
capability of capturing- images from multiple cameras gen- 
erally simultaneously as well as the capability to both 
capture images and be delivering messages or image data 
from the device 330 on a generally simultaneous basis. As 
will be appreciated, the capabilities of the system may be 
increased by the addition of components or enhanced capa- 
bilities of the components which comprise device 330. This 
may include, for example, additional interfaces for digitizing 
image data received from cameras, additional and faster 
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interfaces for input and output devices and increased pro- 
cessing capabilities and data storage to facilitate, enhanced 
function. The required capabilities of device 330 depend on 
the particular type of system that a user desires to operate 
5 and the number and type of connected cameras and other 
devices. 

In the exemplary embodiment shown, a number of dif- 
ferent types of input devices are provided. These input 
devices provide inputs indicative of one or more triggering 

10 events to device 330. Such triggering events cause or may 
affect the manner in which image data is captured by the 
system. Generally the input devices include appropriate 
interfaces in connection therewith to enable the device 330 
to receive signals indicative of the triggering event. The 

1 5 exemplary input devices shown include a cash register 344. 
Cash register 344 which may also be considered a banking 
machine, is connected to device 330 by a communications 
link such as a local network. This enables the device 330 to 
cause images to be captured from a corresponding camera 

20 when signals indicative of transactions are occurring at the 
cash register. It should be understood that cash register 344 
is representative of but one of numerous types of devices 
that may be used in a sales, service provider or banking 
environment and for which it may be desirable to make a 

25 record of activity occurring adjacent to such devices when 
activities are conducted. 

Additional representative input devices include sensors 
schematically indicated 346. Sensors 346 may include sen- 
sors for detecting the opening of doors, windows, ventilation 
ducts or other activities for which it is desired to capture 
images. Another exemplary input device includes an alarm 
input 348. The alarm input 348 may be, for example, a 
device which is actuated by person to indicate an alarm 
condition. This may be, for example, a panic button which 

35 is pressed to indicate a hold-up in a banking or other 
establishment. Alarm input devices may take various forms 
and may include sequences input to computer terminals or 
other devices which are connected to device 330. 

40 Sensors used in connection with the systems may include 
photosensors, infrared sensors, radiation beams or similar 
detectors. Such detectors may be used to sense when a 
person or item passes or occupies a particular space or area. 
For example, a detector may detect when an invisible beam 

45 type sensor is interrupted. Such an invisible beam may 
extend, for example, across a counter or bank teller window. 
As a result, a signal may be given to capture images in 
response to each occurrence of something passing over the 
counter or through the teller window. Similarly, such a beam 

J0 may extend across a cash drawer or similar device. 
Alternatively, such invisible beams may extend in areas 
known only to an employee of the facility. This may enable 
the employee to give a signal to capture images (and perhaps 
activate an alarm) while not making physical contact with 

s5 any device. Numerous systems may be developed using 
these principles. 

Other input devices schematically indicated 350, may 
include other devices which detect or receive indications of 
activity and provide appropriate electrical outputs which can 

so be received by device 330. These may include for example 
heat sensors, infrared sensors, weight sensing pads, elec- 
tronic beams or other types of sensors which can detect 
conditions for which an operator of the system may wish to 
capture images or other data. 

65 In this embodiment of the invention, the cameras them- 
selves may also serve as input devices. The cameras provide 
inputs which enable the detection of certain image condi- 
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tions. Image conditions may include for example, the detec- 
tion of motion within the field of view of the camera. 
Alternative image conditions may include a lack of usable 
video. This may be for example a lack of contrast in an 
image, brightness or darkness beyond selected limits or 
other images or circumstances such as previously discussed. 
Alternatively as previously mentioned, image conditions 
may include the presence within a field of view of persons 
with particular clothing or features, the presence of persons 
with certain body orientations, the presence of certain 
objects such as weapons or the presence of particular types 
of colors or arrangements of colors. Numerous types of 
image conditions which may be determined through analysis 
of the digital images which are available from the cameras 
connected to the system may be used as triggering events. 

In the embodiment of the invention shown, device 330 is 
also connected to output devices. Exemplary types of output 
devices shown include an audible and/or visual alarm sche- 
matically indicated 352. Such an alarm may give persons in 
an area notice of an alarm condition. An alternative form of 
an output device as shown may include lighting devices 
schematically represented 354. Lighting devices may be 
turned on for example in response to programmed sequences 
to illuminate an area where an alarm condition is detected. 
Other types of output devices may include blocking mecha- 
nisms schematically indicated 356. Blocking mechanisms 
356 may operate to block certain areas to prevent access or 
escape. Alternatively in response to some alarm conditions 
as set through sequences programmed in device 330, other 
alarms may cause blocking mechanisms to open to facilitate 
escape of persons from selected areas. Other output devices 
include, for example, communications devices schemati- 
cally represented 358. Communications devices 358 may 
include, for example, police alarms or dial-up devices to 
notify fire or security agencies of alarm conditions which are 
detected. 

As schematically represented in FIG. 28, device 330 is 
connected to a user terminal device 360. User terminal 360 
may be used for providing inputs from users of the system 
as well as outputs to users, as later discussed in detail. 
Device 330 is also shown in connection with a network 362. 
Network 362 like other networks discussed herein, may be 
a communications link suitable for communicating, and may 
be a local network or a plurality of interconnected networks 
through which device 330 is enabled to communicate 
through an appropriate interface. Remote terminals 364 and 
366 are connected to the network 362. The remote terminals 
may be used for providing inputs and outputs to the device 
330. Such terminals may also be used for purposes of 
programming and receiving images from device 330 in ways 
which are later discussed. Other terminals in the network 
may be used to hold data which may be used to identify 
persons, signatures, documents or provide other functions or 
information as previously discussed. 

It should be understood that system 328 is exemplary of 
many system configurations that are encompassed within the 
scope of the present invention. In one exemplary form of the 
invention, device 330 includes a Diebold AccuTrack™ 
digital video system which is commercially available from 
Diebold, Incorporated, the assignee of the present invention. 
Device 330 operates to provide a helpful user interface for 
communicating with and programming the system. Such 
communications may be carried out through the interface at 
a local terminal such as terminal 360 or remotely from 
terminals connected to device 330 through a network such as 
terminals 364 and 366. FIG. 29 shows an exemplary intro- 
ductory screen 368 produced on an output device of a user 
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terminal in connection with device 330. The user terminal, 
like those previously discussed includes a computer with a 
browser operating therein, which can communicate with 
device 330. Screen 368 provides a useful interface for a user 

5 of the system to configure the operation of the system. It also 
provides a useful interface with which users may interact to 
recover and sort images that have been captured by the 
system as well as to carry out other functions. 

Screen 368 as well as other screens presented by the 

10 exemplary device 330 includes a set of icons and indicators 
referred to as a tool bar 370. As shown in greater detail in 
FIG. 30, tool bar 370 includes a plurality of icons 372. Icons 
372 include a home icon 374, a log-off icon 376 and an 
image search icon 378. Other icons included in the tool bar 

15 include a camera check icon 380, a system configuration 
icon 382, a system tools icon 384 and a help icon 386. 
Generally, the icons include an image or representation of an 
object which suggests to a user the function of each. For 
example, the log-off icon 376 includes a representation of a 

20 key that can be turned. The exemplary form of the search 
icon 378 is a representation of a pair of binoculars. Similarly, 
the icon 380 that is selected to conduct a camera check is a 
visual representation of a camera. Each of the icons 372 and 
the functions that a user is enabled to accomplish through the 

25 selection of each is explained in greater detail in FIG. 31. 
The tool bar 370 includes among icons 372 a status icon 
referred to as 388 in FIG. 30. The status icon 388 indicates 
to a user the status of the system. Several status icons are 

3Q provided responsive to the then current status of device 330. 
The various status icons presented in the exemplary 
embodiment are shown in FIG. 32. For example, a visual 
representation of a traffic light showing a green light 390 is 
displayed to indicate that the system is operating to capture 

3j images in the normal manner. A representation of a ther- 
mometer approaching the top of its range is included in an 
icon 392. This icon is displayed to indicate that the storage 
capacity of the data store within device 330 is reaching its 
maximum capacity and is not storing images in the usual 

to manner - 

An alternative icon 394 is displayed to indicate that there 
is a need for a user to exercise caution as the system is 
running with errors. Another icon 396 which is a visual 
representation of a diskette is displayed to indicate that input 

45 changes to the configuration of the device 330 have not been 
applied. An icon 398 which is a visual representation of a 
stop sign is displayed to indicate to a user that an application 
error has occurred or that some other problem has happened 
such that the system is not operating or communicating 

5 0 normally. 

In this exemplary embodiment, a user at a terminal is 
enabled to program or configure operational features of the 
device 330. Preferably a user will be enabled to configure 
many features and operations of the system. This is accom- 

55 plished in the exemplary embodiment by the user making 
selections and inputs from screens or pages in a graphical 
user interface through which a user sets up or changes the 
programming of the system. These interface screens and 
pages are displayed to the user responsive to selection of 

so icons in the tool bar and through subsequent selections as a 
user operates the automated banking machine in response to 
the interface. 

In the exemplary embodiment, one of the aspects of the 
system that a user is enabled to configure is the period of 
6S time that image data and other data including transaction 
data is stored. In this exemplary embodiment, the device 330 
is configured to store data for at least certain programmed 
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periods of time prior to deletion. FIG. 33 shows an exem- 
plary screen 400 which is presented to a user of the system. 
Screen 400 includes image type categories 402. The image 
type categories correspond to the types of triggering events 
which caused an image to be captured. For example, in FIG. 
33 the types of images corresponding to "normal" are those 
images that are captured in response to programmed 
sequences which are done periodically on a routine basis 
such as for a periodic surveillance of an area. Those image 
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are not applied to "normal" images which are those captured 
in response to routine periodic sequences. As represented in 
screen 406, the user may also elect to apply digital signal 
security to no images or all of the images captured in the 
operation of the system. It should be understood that the 
categories of images shown in screen 406 are exemplary and 
in other embodiments other types of image parameters may 
be used. 

A further useful aspect of the exemplary embodiment of 



types which are captured in response to alarms correspond J0 device 330 and the system 328 represented in FIG. 28 is the 
for example to images captured in response to trigger inputs ability of an authorized user of device 330 to program 
such as a panic alarm or an intrusion into a secure area sequences in which images or other information are cap- 
within a facility. Other image types correspond to transac- hired. As is the case in embodiments previously discussed, 
tions. These may include for example in the exemplary sequences include a triggering event and a series of actions 
embodiment transactions conducted at automated banking 15 that are taken by the system in response to a triggering event, 
machine 332 or at cash register 344. Through inputs in Triggering events may include, for example, sensing image 
response to screen 400 a user is enabled to input and select conditions such as motion, lack of usable video or a blocked 
which types of images are to be deleted first and last. The camera and taking a series of actions in response thereto 
user is also enabled to set up minimum periods during which such as capturing images from other cameras, turning on 
images corresponding to particular image types are to be M lights, placing in permanent storage temporary image data 
retained. that was captured prior to the triggering event, sending 

FIG. 34 shows an expanded screen 404 which further messages such as e-mails or performing other actions, 
enables a user to configure the auto deletion feature of the Similarly, triggering events may include activities conducted 
invention. Through inputs in response to screen 404 a user at an automated banking machine or other transaction 
is enabled to set the unit to accomplish automatic deletion of ^ machine, during which times it is desirable to capture and 
images in accordance with the parameters that have been permanently retain images from cameras which have a field 
input. The user is further enabled to input when the auto 
deletion activity is to begin as well as when available disk 
space is considered sufficient such that auto deletions should 
stop. As a result in response to the user selecting to have auto 3 
deletion activity occur, the device 330 will operate to 
selectively begin deleting images in accordance with the 
priorities that have been established for the retention of 
images so that additional storage space may be made auto- 
matically available. 3 

It should be understood that the parameters and deletion 
capabilities shown in connection with screens 400 and 404 
are exemplary and other embodiments of the invention may 
operate to store image data and delete it selectively ii 



response to other parameters. In addition, the auto deletion 40 1031 are carried 



of view that includes the area where the machine is posi- 
tioned. Similarly triggering events may include inputs to or 
from alarms or sensors. Other triggering events may include 
sequences which operate on a timed or other periodic basis 
in a routine manner such that image data is stored in 
relatively permanent storage from each of the cameras in the 
system as a routine matter of course. Numerous types of 
sequences can be programmed by an authorized user using 
the exemplary embodiment of the invention. 

For purposes of the particular exemplary embodiment of 
system 328, triggering events are cataloged by type as either 
"normal", "alarms" or "transactions". Normal images are 
those that are captured in accordance with routine sequences 



function may be combined or integrated with an automated 
downloading function so as to selectively transfer images 
prior to deletion to another storage area that is connected to 
device 330. This may include, for example, the transfer 
image and transaction data to other terminals connected in 
network 362 so that such image data may be stored at a 
remote location prior to deletion of the image data from the 
device 330. Other approaches and techniques appropriate for 
systems of the invention will be apparent to those skilled in 
the art from the foregoing description. 

Another aspect of the exemplary embodiment that may be 
configured by an authorized user is the security applied to 
various types of images. In the exemplary embodiment 
device 330 allows a user to selectively apply authenticating 
algorithms to selected types of images. Ascreen presented to 
a user in the course of configuring the system to establish 
this capability is represented 406 in FIG. 35. In response to 
screen 406, a user is enabled to set the system so that digital 
signatures are applied to any of several different image 
types. For example as represented in screen 406, a user may <so 



a periodic basis in accordance with the 



programming of device 330. Different r 
may be operative at different dates and times in accordance 
with the system configuration. Such routine sequences may, 
for example, capture an image from a particular camera so 
45 as to store it in relatively permanent memory every so often, 
then subsequently capture an image from another camera 
and so on. Because these "normal" images are captured on 
an ongoing basis, care is generally exercised by the operator 
of the system to be sure that not so many images are stored 
50 that the available storage space is occupied too quickly by 
images that are of no particular interest. 

The images classified as "alarms" are those that corre- 
spond to alarm type inputs. These can include hard trigger 
alarms such as those provided by switches, invisible beams 
5S and buttons that may be tripped as activities occur. Similarly, 
the category of "alarms" include image conditions such as 
motion detection, loss of usable video, detection of particu- 
lar features, clothing, body orientation, colors or objects 
within the field of view (or a detection area smaller than an 
:rall field of view) of a particular camera. Each alarm 



elect to include digital signatures in images captured in sequence may include appropriate actions such as actuating 

response to triggering events such as alarm conditions, lights, blocking devices, alarms, contacting police or fire 

detection of motion or other hard trigger alarms. Likewise as departments and/or sending e-mail messages and/or images 

shown in screen 406, the user may configure the system to to predetermined addresses. 

apply digital signatures into images captured in response to 65 In the exemplary embodiment, images associated with 

transactions conducted at an automated banking machine. In "transactions" are images associated with devices at which 

the particular example, shown in FIG. 35, digital signatures transactions are carried out. These may include 
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conducted at automated banking machine 332, cash register 
344 oi other devices where it is desirable to make a record 
of the transactions. With regard to transaction images the 
sequence typically involves a triggering event related to 
operation of a component of a transaction function device or 
terminal, and the actions may include capturing the image to 
store it in memory and perhaps additional steps depending 
on the nature of the transaction being conducted. Again, it 
should be remembered that the categories of triggering 
events in this embodiment are exemplary and other trigger- 
ing event categories may be used in embodiments of the 
invention. 

In the exemplary embodiment, device 330 operates in a 
manner like that previously discussed to digitize image data 
received from all or a selection of cameras on an ongoing 
basis. This image data is digitized as image frames on an 
ongoing basis and Temains stored in the memory associated 
with the computer of device 330 for a fairly limited period 
of time. These temporarily captured and stored images may 
be more permanently captured by being moved to relatively 
permanent storage at the time that they are received or 
alternatively may be moved into relatively permanent stor- 
age at any time prior to their deletion. The value in digitizing 
and temporarily capturing images on an ongoing basis as 
often as possible from selected cameras include the ability to 
recover image data from a time prior to a triggering event. 
Thus for example if an image condition such as a blocked 
camera is detected, one or more prior images from the same 
camera that are still in temporary storage may be transferred 
l response to the triggering event to more permanent 
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the device 330. These outputs may include for example a 
descriptive name for the particular item or action which is 
triggered by the output. For example, in FIG. 37 there is 
shown a screen 410 in which an authorized user of the 

s system is presented with output numbers for the various 
contacts and connections that may be made to device 330. 
By making an appropriate selection and input, the user is 
enabled to apply descriptive terminology to these outputs. 
For example, in screen 410 "output 01" has been named to 

to indicate that it operates to turn on lights in a vestibule. Of 
course this is exemplary and any appropriate name may be 
input in the discretion of the operator. 

Similarly, the user of the system is enabled to provide 
descriptive names for the inputs which serve as triggers for 

15 executing sequences by device 330. Screen 412 in FIG. 38 
shows the capability of a user to give a descriptive name to 
a particular input device. For example, in FIG. 38 "input 01" 
to device 330 is indicated as associated with a teller panic 
button. Having this descriptive information available and 

20 usable to program sequences in the invention makes it much 
easier for a user to set up and check that the desired activity 
is happening in response to a triggering event in any given 
sequence. 

FIG. 39 also shows the capability of device 330 to execute 
sequences that are triggered by operation of automated 
banking machine 332. A screen 414 in FIG. 39 shows an 
ATM monitoring set-up screen. In response to the presen- 
tation of screen 414, a user is enabled to give the automated 



storage and correlated with data representing the triggering 30 baQkm g machine a particular descriptive name. This 

. tl: . descriptive name may include the particular street address 

where the automated banking machine is located. Similarly, 
if there are several automated banking machines at the same 
address, each machine may be assigned a descriptive name 
representative of its location. Such terms used may include 
names such as "lobby ATM", "drive-through ATM" and 
"walk-up vestibule ATM". Of course many other types of 
names and designations may be used depending on the 
particular type of automated banking machine involved. 

In the exemplary embodiment of screen 414 shown in 
FIG. 39, the system 330 is shown configured to operate in 
connection with an ExpressBus™ interface which is used in 
automated teller machines manufactured by Diebold, 
Incorporated, the assignee of the present invention. Of 



it. This may enable detection of an image which includes 
a person who caused a camera to be blocked. The ability to 
retain on a more permanent basis images which occurred 
prior to a triggering event is configurable in the system, as 
are the number of images prior to the triggering event which 
may be transferred to more permanent storage. Of course the 
ability to transfer prior images depends on the number of 
image frames that are available in temporary computer 
storage from each camera prior to the deletion of such 
images. Of course the duration that such temporary images 
are stored can be increased with the addition of additional 
storage and processing capability. Likewise, the frequency 
of these temporary images from any given camera depends 
on the processing capabilities of the computer operating ' 



device 330. Faster processing may similarly increase the A5 course in other embodiments of the invention other appro- 
frequency at which temporary images are captured. priate set-up screens suitable for configuring the program- 
A useful aspect of the exemplary embodiment includes min S of ^ c system to work with other types of machines 
the ability to program sequences using descriptive terminol- mav De presented. 

ogy which is established by a user of the system. FIG. 36 As was discussed in connection with other embodiments, 

shows a screen 408 that is displayed to a user in configuring 50 actions performed as part of a sequence may include sending 

system 328. Screen 408 is a camera set-up screen in which e-mails to one or several persons notifying them of the 

a user is enabled to give descriptive names to the particular occurrence of the triggering event. Screen 416 shown in 

cameras or field of view of a camera connected to the FIG. 40 may be used by an authorized user of the system to 

system. From screen 408, a user is enabled to select a camera input e-mail addresses that are to be notified of triggering 

through 1 use of an input device such as a mouse and to "sec" 55 events. Further as represented in screen 416, a user is able 

associated therewith. The user is also to designate groups of persons who are to be notified of 



the field of' 

enabled to input a descriptive name for that field of 
such as is shown in connection with "camera 01" shown in 
FIG. 36. As subsequently explained in detail, a user is 
enabled to configure sequences including triggering events 
and actions to be taken in response thereto using the descrip- 
tive names that the user has given to various cameras in the 
system. This capability greatly simplifies the programming 
of the system as users are not required to leam any special 
conventions or terminology. 

As is the case with cameras, users are also enabled to 
apply descriptive 



particular events. These descriptive names for groups enable 
an authorized user to readily configure the system so that a 
number of people receive an e-mail message notifying them 
60 of a triggering event. Such actions are readily programmed 
into a sequence by referring to the name of the group. 

Screen 418 shown in FIG. 41 shows an example of an 
e-mail group which has been named "security". This would 
be, for example, a group of persons or entities that are to be 
65 notified in the event that a triggering event detected by the 
system indicates a breach of security or some activity that 
jutputs which are provided from should be investigated by a security organization responsible 
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for the facility. As can be appreciated by screen 418, an 
authorized user of device 330 is enabled to add, delete and 
edit e-mail addresses which compromise the groups which 
arc to be notified. 

The exemplary embodiment of device 330 enables an 
authorized user to readily program the system to carry out 
various types of sequences. These sequences include 
sequences associated with the capture of "normal" or routine 
images that are stored on a timed or other periodic basis 
while the system is operating. The user is also enabled to 
program sequences in response to the various types of 
triggering events such as inputs, motion detection, lack of 
usable video and the conduct of transactions. FIG. 42 shows 
a screen 420 which is presented to a user in connection with 
establishing routine sequences for the capture of images and 
storage on a relatively permanent basis. Screen 420 also 
shows the beginning point for the programming of 
sequences in response to input devices which will be later 
discussed in detail. 

In response to user selection of the "daily program" box 
in screen 420 a screen 422 shown in FIG. 3 is presented. 
Screen 422 shows a visual representation of a weekly layout 
for hours in each day and the names of programs or 
sequences which are operated to capture routine or "normal" 
images during the indicated times. In addition to viewing the 
sequences that are operative at various times during the 
week from screen 422, the user is enabled to view the 
sequences applicable during any selected day of the week or 
in groups of days such as by weekdays or weekends. In the 
exemplary programming of the system, sequences are con- 
figured to continue until a time when another sequence is to 
be initiated. Further, the programming is set up so that a 
more specific program for a given time period will override 
a more general program during the selected period. 

By selecting a particular day of the week from screen 422, 
the exemplary embodiment of device 330 is operative to 
display to a user a screen 424 shown in FIG. 44. This screen 
shows for the selected day the sequences for the routine 
capture of images that occur on that day and the time periods 
when each sequence starts and ends. A graphical indication 
is also provided so that the user may readily see the times 
during the selected day when particular sequences are opera - 

From screen 424, a user is enabled to select to view any 
of the selected sequences. For example, by selecting to view 
sequence indicated "1" in screen 424 the device 330 causes 
the screen 426 shown in FIG. 45 to be displayed. Screen 426 
indicates to the user a graphical representation of the steps 
involved in the routine sequence. Screen 426 also indicates 
the data compression level that is to be applied to the images 
that are captured and stored on a relatively permanent basis. 
By selecting the compression level the user may choose to 
have lower quality images in exchange for utilizing less of 
the available data storage space with images corresponding 
to the particular sequence. Various levels of data compres- 
sion are selectable by the user for the sequence as shown in 
screen 426. 

As represented in screen 426, the user is also enabled to 
set an image capture rate which controls the frequency of 
image capture and storage during time periods which are 
indicated in the sequence as periods during which images 
are to be captured by a particular camera. In the exemplary 
embodiment, the user has the option to capture a certain 
number of images or to set the system to capture images for 
a period of time. If the user configures the system to capture 
images based on time, the indicated rate reflects the number 
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of images captured and stored in relatively permanent 
memory during each second. The exemplary embodiment 
also enables a user to select AVI which is an image capture 
rate sufficiently high such that it appears to capture full 
S continuous motion in a manner similar to a video clip. In the 
exemplary embodiment the capture of ten or more images 
each second corresponds to what generally appears to a user 
to be full motion. Of course, higher rates of image capture 
may be used. 

10 Screen 426 represents the sequence which is carried out 
routinely by the system on an ongoing basis using the 
passage of time as the triggering event for each sequence. As 
can be seen, the particular cameras in the exemplary 
sequence are shown by the numbers as well as the descrip- 
15 tive names which have been applied by a user thereto. In this 
exemplary sequence, a camera which views a front door 
takes one image every second for three seconds. Thereafter, 
a camera which takes pictures of an outside ATM takes one 
image every second for three seconds. After that, a camera 
20 which views the back door takes one image every second for 
three seconds. After completing the sequence, the sequence 
repeats. An authorized user is enabled to modify the 
sequence by changing the number and timing of images in 
the sequence. The user is also enabled to delete and modify 
25 steps in the sequence by selecting the "buttons" at the 
bottom of screen 426. For example, a user is enabled to 
selectively or completely delete steps in the sequence, add 
cameras, add steps and save the revised sequence. Of course 
in other embodiments additional options for steps or actions 
30 in sequences may be provided. 

In the exemplary embodiment of the routine sequences, 
provisions are not made for notifying a remote location via 
e-mail. This is because routine sequences are continuously 
■ executed gathering and storing images at all times while the 
35 system is operating. This includes times in which images are 
being captured in response to other events. In other embodi- 
ments however, the system could include as part of the 
capture of normal images, provisions for providing periodic, 
reports via e-mail or otherwise, to functions or individuals 
4 0 who need to know that the system is operating normally. In 
addition, such messages may also include one or more 
images enabling the person receiving the message to visu- 
ally verify the current condition in the area or facility 
monitored by the system. 
45 FIG. 46 shows a screen 428 at which a user is enabled to 
review sequences associated with inputs that correspond to 
triggering events. Such triggering events may include, for 
example, the inputs from various sensors sensing activity in 
various areas under surveillance, inputs from panic buttons 
50 or other types of inputs. By providing inputs in response to 
screen 428 an authorized user is enabled to selectively 
enable execution of the sequences in response to the trig- 
gering events which cause the listed inputs. 

By providing inputs to screen 428 a user is enabled to 
configure the sequences associated with particular inputs. 
This includes establishing a schedule during which a 
selected input will cause a selected sequence to be executed. 
The schedule for the execution of a particular sequence is 
shown in screen 430 in FIG. 47. Through inputs to screen 
430, the user is enabled to indicate the time periods during 
which the system will execute the sequence if the input is 
received. For example, if the particular input is associated 
with opening a door, it may not be desirable to capture 
images during the time periods when the door is frequently 
opened by employees or customers who access a facility. 
The configuration associated with the input enables the input 
to cause the execution of the sequence only at the times 
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when the capture of images is likely to yield useful infor- 
mation. In the exemplary screen 430 shown in FIG. 47 an 
input number 2 is configured to cause its corresponding 
sequence to be executed only between 9 a.m. and 4 p.m. 
Through inputs to screen 430, an authorized user is enabled s 
to modify these time periods as well as to select separate 
discrete times periods during which the input will cause the 
sequence to be executed. 

The user is also enabled to set up or modify the sequence 
that is associated with the input. This is accomplished from 10 
screen 430 by an appropriate input that causes the screen 432 
shown in FIG. 48 to be displayed. Screen 432 includes a 
description of the particular event which is associated with 
the input. Also as is the case with the routine sequences 
previously discussed, a user is enabled to set the image 15 
quality of the images captured and stored in response to the 
triggering event Further in the exemplary embodiment, an 
authorized user is enabled to set the number of times that the 
sequence will be executed in response to the triggering 
event. As previously discussed screen 42 also includes M 
provisions for the user to set the image capture rate associ- 
ated with the capture of images that are done in the corre- 
sponding sequence on a timed basis. 

The user is enabled to set up a sequence by selecting the 
"buttons" at the bottom of screen 432. These buttons cor- ls 
respond to various actions related to cameras, outputs and 
e-mails that the system is enabled to capture images from, 
provide and send, respectively. In response to selecting one 
of these buttons, a particular configuration step or action 
which a user may populate with instructions by making 30 
selections therein is included in the sequence. For example, 
in response to selecting a "camera" button a sequence frame 
designated 434 in screen 432 is displayed. The sequence 
frame includes five areas for inputs that can be provided by 
the user. This includes the camera selection, the number of 35 
images, the frequency of the images and the duration or 
number of images involved. By populating these five spaces 
in the image frame with data the user is enabled to provide 
the necessary programming information for carrying out an 
action in a manner that is readily understood in a sentence 40 
format. For example, as shown in screen 432, sequence 
frame 434 indicates that the camera designated "drive-thru 
#2" takes one image every one second for two seconds. Of 
course by making selections and inputs the user is enabled 
to change the five input areas within the sequence frame to 45 
suit their particular requirements. 

Similarly, as represented in screen 432 selection of the 
"output" button enables a user to include a sequence frame 
436 in an action the sequence. The sequence frame includes 
three inputs that can be made by a user to select the nature 50 
of the output that is to be included as an action in the 
sequence. In the case of sequence frame 436 the user is 
shown as having populated the information for causing the 
"W station #2 light" to turn on for ten seconds. Thus again 
the sequence frame enables the user to provide in a sentence 55 
format those instructions which correspond to a selected 
output. Further the outputs are enabled to be selected in 
accordance with the descriptive names that have been 
applied to the outputs by a user. 

As can be appreciated from screen 432 numerous action 60 
steps can be selectively added or deleted from a given 
sequence as desired by the user in response to the triggering 
event. It should further be understood that similar sequence 
frames are provided for e-mails which is a selected action 
step that can be taken in response to a triggering event. 65 
Further in other embodiments additional types of steps can ' 
be taken, each of which may have its own sequence frame 
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which a user may populate with particular data to accom- 
plish the carrying out of a particular action step. For 
example, additional actions may include repeating one or 
more steps in a sequence one or more times, and waiting for 
other actions or delaying for a time before taking further 
actions. Similar principles are carried out in connection with 
the programming of the various types of sequences by the 
system of the exemplary embodiment. 

FIG. 49 shows a screen 438 which is associated with 
establishing a sequence in response to the detection of 
motion in the exemplary system. The motion set-up 
sequences enable a user to' establish when detected motion 
within a particular area causes images to be captured and 
stored on a relatively permanent basis, and other actions to 
be taken as part of a sequence. 

In screen 438, the cameras which are included in the 
system are presented using the descriptive naming termi- 
nology applied by a user. In response to the motion set-up 
screen 428 a user is enabled to select which sequences are 
enabled or disabled for particular cameras. In addition a user 
is enabled to access other screens for purposes of setting up 
selected detection areas in which motion is to be detected, as 
well as to configure the sequences that are executed in 
response to motion detection. 

In response to selecting a set-up button for an appropriate 
camera from screen 438, a setup screen showing a field of 
view currently obtained from the camera selected is dis- 
played at the user terminal. An example of such a set-up 
screen is indicated for 440 in FIG. 50. 

Screen 440 includes a field of view of the designated 
camera generally indicated 442. The field of view of the 
camera includes the entire image field that the camera is 
currently viewing. Through use of a mouse or other input 
device, a user is selectively enabled to select one or more 
detection areas schematically indicated 444 within the field 
of view 442. The detection areas 444 are one or more areas 
to be analyzed and in which a determination concerning the 
detection of motion is to he made. An advantage of provid- 
ing a selected detection area for purposes of determining the 
presence of motion is that it avoids problems associated with 
monitoring in areas where motion may commonly be occur- 
ring in some areas, but where in other areas the occurrence 
of motion is an event for which images should be captured. 
In the exemplary embodiment the system is operative to 
compare the images only within the selected detection areas 
on an ongoing basis between the temporary captured images 
that are stored temporarily from each of the cameras. Com- 
parison of the image in one or more successive ones of these 
temporary images are preferably analyzed through operation 
of the computer for differences. In this embodiment the 
computer operates to analyze the pixels which make up 
these images for a degree of change. If more than a set 
degree of change between one or more of these images 
which are spaced in time is detected, this is an image 
condition indicative and motion and a triggering event 
which causes the corresponding sequence to be executed. 

An advantage of the exemplary embodiment shown in 
connection with screen 440 is that the user is enabled to 
selectively set the degree of change in the image in the 
detection area which will result in a determination that 
motion has been sensed. Specifically in the exemplary 
embodiment the user is enabled to selectively input values as 
to a percent of sensitivity which corresponds to a change in 
property such as intensity or color (or a combination of both) 
among pixels in the detection area that will be considered for 
purposes of determining whether motion has occurred. Like- 
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wise the user is enabled to set the percent of activity which 
corresponds to a quantity such as a number or percentage of 
pixels subject to analysis experiencing the set change in 
intensity or sensitivity which is indicative of motion. In this 
way the user of the system is enabled to set the motion s 
detection parameters for the degree of change which will 
cause a triggering event indicative of motion detection. A 
user may thereby avoid motion from being considered 
detected in circumstances where it is not desirable to capture 
images. 10 

An exemplary embodiment of the invention includes a 
service program which enables a servicer or authorized user 
to test the suitability of the motion detection settings in 
particular circumstances. This program runs in one or more 
computers operatively connected to the camera of interest. 1S 
The user inputs into the computer running the program the 
selected sensitivity and activity settings. The user may then 
cause activity to occur in the field of view of the camera. The 
program then causes a display to operate so as to indicate 
whether the activity resulted in motion being considered to 20 
have been detected. In this way a user may adjust the settings 
to suit their requirements. Alternatively the system may be 
operated in a test^mode to capture a series of images from a 
selected camera. The settings may be applied by a test 
program to these captured images in a controlled manner to 2 s 
evaluate the settings versus the nature of image change. In 
an exemplary embodiment, captured images may be com- 
pared in the sequence originally captured or may be com- 
pared in a different sequence to determine the appropriate 
motion detection settings. Once selected, the selected set- 30 
tings for sensitivity and activity may be set in the system and 
applied on an ongoing basis. 

Returning to the discussion of FIG. 50, from screen 440 
a user is enabled to display a schedule for selected days in 
which motion is to be detected. This is represented in screen 35 
446 which is shown in FIG. 51. Through inputs responsive 
to screen 446 the user is enabled to set the periods during 
which motion detection is accomplished for purposes of 
carrying out a sequence. As can be appreciated in many 
circumstances there are particular times of day during which ao 
motion is likely to be going on in a particular area and other 
times during which the detection of motion may represent an 
usual event for which images should be captured. Through 
inputs of screen 446 an authorized user is enabled to 
selectively set the times during which motion detection 45 
analysis will be conducted. 

From screen 446 a user is enabled to set the sequence that 
is carried out in response to a motion detection event. This 
is done in response to a screen 448 shown in FIG. 52. Screen 
448 includes the ability of a user to set the parameters 50 
associated with the detection of motion using the descriptive 
names for cameras which were set-up by the user. The user 
is also enabled to set the image quality parameters for the 
storage of images. In addition to parameters associated with 
other screens, in screen 448 the user is also enabled to set the 55 
number of images captured prior to the detection of motion 
which will be moved from temporary storage into relatively 
permanent storage in connection with images captured in 
response to the motion event. Using inputs directed to the 
"buttons" in screen 448, the user is also enabled to set up the 60 
sequence frames associated with cameras, outputs and 
e-mails by populating the information in the frame. A 
sequence frame enables the user to program using a sentence 
type structure, the actions which will occur in response to 
the triggering event. For example, in the sequence shown in 65 
448 in response to motion being detected at the camera 
which watches the back door of a particular facility, the back 
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door camera takes two images every second for sixty 
seconds. Thereafter the outside back light turns on for five 
seconds. In addition to capturing the images from the back' 
door camera, two prc-alarm images are transferred from 
temporary storage into relatively permanent storage with 
data which describes the triggering event. Of course, it 
should be understood that the sequence parameters and 
actions are exemplary and in other embodiments other 
approaches may be used. 

Embodiments of the invention also capture images in 
response to triggering events which are indicative of cam- 
eras being blocked. Such blocked camera events which are 
alternatively referred to herein as a lack of usable video, 
generally result from an image condition in which the image 
presented is either unduly light or dark, or otherwise lacking 
in contrast, not changing or otherwise appearing so as to 
suggest that usable video data is not being received. The 
sequence as associated with blocked cameras is configured 
in the exemplary embodiment with inputs responsive to a 
screen 450 shown in FIG. 53. In response to presentation of 
the screen 450 a user is enabled to select the particular 
camera at which a blocked camera event will be detected. 

In response to the user selecting a camera in response to 
screen 450, the exemplary embodiment displays a screen 
452 shown in FIG. 54. Through selections made in response 
to the presentation of screen 452 the user is enabled to set the 
blocked camera capability as either operative or inoperative. 
The user is also enabled to set up the criteria used for 
identifying a blocked camera as a triggering event and to 
configure the sequence that will be executed in response to 
the blocked camera event. 

In response to a user selecting the set-up button from 
screen 452, the exemplary embodiment is operative to 
display a screen 454 shown in FIG. 55. In screen 454 the 
user is enabled to set a brightness intensity (which may 
represent a color level tending toward white) as well as a 
darkness intensity (which may represent a color tending 
toward black). In this exemplary embodiment if the pixels 
which make up the field of view of a selected camera 
average above the selected brightness intensity, or alterna- 
tively average below the selected darkness intensity, a 
triggering event indicative of lack of usable video is initi- 
ated. Alternative embodiments may look for every pixel 
being above or below certain thresholds. Alternatively in 
other embodiments of the invention the pixels which make 
up the field of view are analyzed by the computer on an 
ongoing basis for color level or contrast with pixels in other 
areas of the field of view. A failure of the image to have 
contrast above a set level for the overall field of view may 
in addition represent a triggering event indicative of lack of 
usable video. Of course, as previously discussed, other 
criteria may also be used for deciding that there is a lack of 
usable video. 

Screen 456 shown in FIG. 56 is presented to a user in the 
exemplary embodiment to set a time period during which the 
sequence will be carried out if a camera is blocked. The user 
is enabled to set the inputs for those times of day during 
which a blocked camera event will be considered a trigger- 
ing event for the sequence to be carried out. 

FIG. 458 shown in FIG. 57 is displayed in the exemplary 
embodiment to a user to configure the sequence that is 
executed in response to a blocked camera event. As in the 
other sequence configuration screens of the exemplary 
embodiment, a user is enabled to set the quality of the image 
data that is captured in response to the triggering event. 
Further the selection of "buttons" in the lower portion of the 
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screen 458 causes sequence frames to be displayed which 
the user is enabled to arrange and populate with data to 
configure the sequence. As shown in FIG. 58 the sequence 
frame 460 associated with sending e-mails is displayed. This 
sequence frame enables a user to input data identifying 5 
persons or groups of persons to which e-mails are to be sent. 
The ability to use the descriptive naming terminology 
defined by the user simplifies the programming of the 
sequences in the exemplary embodiment. Further as shown 
in screen 458 the user is enabled to employ other sequence 10 
frames such as sequence frame 462 which is associated with 
a camera. By populating the inputs for the camera sequence 
frame the user creates a sequence which is carried out in 
response to the indicated camera being blocked. The exem- 
plary sequence includes sending an e-mail to the e-mail 15 
group that is designated "security". In addition to sending 
the e-mail, camera #2 is operated by the computer to capture 
and store two images every second for twenty seconds. Of 
course it should be understood that camera number two is a 
camera which preferably includes in its field of view the 2 o 
camera that is indicated as blocked. Of course as previously 
explained in other embodiments, the programming for lack 
of usable video may also include the retention in more 
permanent memory of temporary images which were takerj 
by the blocked camera prior to the lack of usable video being 25 
detected. Such images may indicate the cause of the lack of 
usable video. Of course other approaches may be used in 
embodiments of the invention. 

In the exemplary embodiment of the invention device 330 
is also configured to execute sequences in response to 30 
triggering events such as transaction steps which occur at 
automated banking machine 332 or cash register 344. In the 
exemplary embodiment sequences are configured to acquire 
images in response to the operation of transaction function 
devices. The images are stored in connection with transac- 35 
tion data regarding the transaction that is conducted at the 
machine. FIG. 58 shows an exemplary screen 464 which is 
displayed to an authorized user by device 330 in connection 
with configuring sequences responsive to the operation of an 
automated banking machine. Through inputs in response to aq 
screen 464 a user is enabled to set up and configure the 
sequences associated with operation of the machine. 

In the exemplary embodiment of the invention inputs 
responsive to screen 464 enable the user to set up the 
acquisition of images from particular automated banking 45 
machines. This is done through inputs to the user terminal in 
response to a screen such as screen 414 shown in FIG. 39. 
Further from screen 464 a user is enabled to configure the 
parameters for the acquisition of images in connection with 
particular events carried out at the ATM. This is accom- 50 
plished in the exemplary embodiments through inputs 
through a screen 466 shown in FIG. 59. Screen 466 enables 
a user to select triggering events for the capture of images. 
For example in the exemplary screen shown, the triggering 
events include the reading of a user's card and the printing ss 
of a receipt. The user is also enabled to configure the system 
to set the quality of the images stored based on the level of 
data compression used. Further as represented in screen 466, 
the user is also enabled to set sequences which include 
sequence frames for cameras, outputs and e-mails respon- 60 
sive to triggering events which occur in the course of a 
transaction. For example in exemplary embodiments of the 
invention the system may capture one or more images of a 
customer operating the banking machine so as to provide 
verification at a later date as to toe identity of the particular 65 
person who has operated the machine to carry out the 
transaction. The number and character of images may be 



varied depending on transaction parameters including the 
transaction type, the time of day, the amount of money 
involved or other parameters associated with the user. 

In the exemplary embodiment of the invention, transac- 
tion data is also stored in correlated relation with the 
captured image data. The image data is correlated with the 
transaction data by the particular time at which the transac- 
tion is conducted. Of course in other embodiments other 
approaches to correlating image and transaction data may be 
used. Alternatively, image and transaction data may be 
stored together in common files depending on the require- 
ments of the system. Generally, in the case of a system 
monitoring an automated banking machine, the transaction 
data that is stored will include parameters such as time, user 
name, account number, transaction type and amount. The 
storage of these parameters in connection with the images 
enable the selective sorting of images by transaction param- 
eters. This enables an operator of the system to more readily 
recover types or ranges of transactions that may of interest. 
In addition, it facilitates the selective retention or deletion of 
transactions in some embodiments by factors such as the 
transaction type and/or amount. Of course, in other embodi- 
ments other approaches may be used. 

It should be understood that although in the exemplary 
embodiment image capture from an automated banking 
machine is conducted responsive to signals sent to transac- 
tion function devices on the system bus of the ATM, in other 
embodiments other triggering events may be used. For 
example, in embodiments of the invention the presence of a 
user adjacent to a transaction terminal may be sensed with 
a proximity sensor such as an ultrasonic detector or a weight 
sensing pad. Alternatively, automated banking machines 
may provide hard sensor inputs such as are obtained when 
a user receives cash from a cash receipt dispenser, or another 
device on the machine is moved. Such inputs may be 
configured as inputs to device 330 much in the manner of 
sensors 346. Such inputs may be used for purposes of 
configuring sequences in response to such inputs. For 
example a screen 468 shown in FIG. 60 represents an 
example where an input from a sensor is used as the basis for 
configuring a sequence. Such an input may correspond to the 
operation of the device on an automated banking machine or 
other transaction terminal. Through inputs responsive to 
screen 468 a user is enabled to configure a sequence includ- • 
ing capturing images from cameras, providing outputs or 
sending e-mails in response to such inputs. Of course, 
numerous other alternatives for accomplishing similar func- 
tions are encompassed within the scope of the present 
invention. 

As previously discussed, a useful aspect of this form of 
the present invention is the ability to conduct searches for 
images that have been stored. Searches may be made based 
on one or more parameters. Image searching is accom- 
plished responsive to selecting the icon 378 in the tool bar 
370 displayed on numerous screens in the exemplary 
embodiment A screen 470 shown in FIG. 61 is used for 
obtaining user inputs concerning exemplary parameters that 
are employed in the searching of images. As can be seen in 
screen 470 a user is enabled to select time periods during 
which images are to be searched. The user is also enabled to 
select cameras which captured the image data which will be 
searched. The user is enabled to select all cameras or 
particular cameras which are to be searched. Alternatively, a 
user is enabled to select a "quick viewer option" which 
enables a user to scan through images in a manner that is 
later described. 

Screen 470 also enables a user to select parameters for 
identifying images. These include for example selecting 
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among images captured in response to particular alarm types 
as well as images captured in response to particular trans- 
action types. In this way a user is enabled to selectively 
search the images for a number of different parameters. The 
ability to search by various parameters greatly reduces the 
effort required to identify images that may fit a user's search 

As explained in connection with other embodiments, 
image data may in addition be searched by visual charac- 
teristics. These may include for example searches for physi- 
cal characteristics of persons shown in the images. Similarly 
searches may be made for certain types of apparel, certain 
colors or certain devices or items. The capability of the 
embodiments of the present invention may enable identify- 
ing particular persons or situations for which available data 
is otherwise incomplete. This may include for example 
identifying witnesses or other persons present when particu- 
lar circumstances have arisen. Of course numerous different 
types of criteria and parameters may be used in searching for 
selected images. 

A further aspect of the exemplary embodiment repre- 
sented in screen 470 is the ability to also group images by 
the particular type of event which has caused the images to 
be captured. This provides the capabilities of allowing a 
person reviewing images to catalogue and review images by 
the triggering event- which caused them to be captured 
together. This may provide a user with additional insights as 
to particular events. It may also enable a user to search an 
event type of most interest first before moving onto other 
images which meet search criteria. 

In response to the conduct of searches, various image 
pages are displayed by the device 330. Examples of image 
pages are shown in FIGS. 62 through 72. Each of these 
image pages shows one or more images that have been 
captured and stored, and which are displayed in response to 
search requests. The nature of each of the image pages and 
how they are used in the exemplary embodiment are 
explained in detail in the charts shown in FIGS. 73 and 74. 
Of course it should be understood that in other embodiments 
of the invention other image pages may be used. 

It should be noted that in the exemplary embodiment of 
the invention, a control panel schematically indicated 472 is 
displayed in connection with image pages. A control panel 
472 enables a user to perform various functions to review 
images, as well as to download images from device 330 to 
other terminals in the system with a greater degree of 
assurance that the images have not been tampered with. It 
should also be noted that in image pages of the exemplary 
embodiment a graphical representation of a piece of movie 
film is included to represent to a user that a series of images 
were acquired at high frequency in response to. an event so 
as to acquire a film clip that approximates full motion video. 

A further aspect of the present invention that is useful is 
the ability of the system to indicate that a plurality of images 
have been captured in response to certain triggering events. 
This is indicated by the image sets as represented for 
example in FIGS. 64 and 65. Further as represented for 
example in FIGS. 67 through 70, particular images may be 
- selected for enlargement for review by a user with informa- 
tion concerning the nature of the triggering event which 
resulted in the capture of the image. A listing of the data 
which is included with triggering events and which can be 
recovered by an authorized user of the system is listed in the 
chart in FIG. 75. A further useful feature of the exemplary 
embodiment is the capability of a user to provide comments 
concerning particular images. Such comments may be input 
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from the user terminal via typed input in text form. In 
alternative embodiments, a user may input comments by 
voice to text conversion input as well as to have comments 
stored as a voice file. Such comments may be useful later in 

5 recovering images when searching by particular comment 
criteria. The computer may itself be programmed to add 
comments to particular fields in connection with images 
depending on the programming of the system. 

The control panel 472 used in the exemplary embodiment 

10 is shown in greater detail in FIGS. 76 through 80. The 
control panel 472 includes a plurality of icons and indicators 
as well as an image counter which is shown in FIG. 77. The 
function executed in response to selection of each of the 
icons in the control panel when particular image pages are 

15 being displayed is shown in detail in FIGS. 78 through 80. 
As will be understood from the detailed description, the 
control panel 472 enables a user to navigate through images 
in a rapid and selective manner. The user is also enabled to 
navigate through a series of images sequentially in varied 

20 increments and directions within the series of selected or 
displayed images. Further as represented in FIG. 80, the user 
is enabled to provide inputs to the control panel so as to 
identify images captured within certain time parameters. It 
should be understood that in embodiments of the invention 

25 the series of images may be considered to be one dimen- 
sional. However in other embodiments the images may be 
arranged in a matrix or other multilayer or multidimension 
format based on varied parameters. By making selections 
and inputs users may navigate in varied directions in the 

30 series of images. 

FIGS. 81 through 83 show numerical examples of the 
capability of the control panel 472 in enabling a user to 
navigate through a series of images which arc displayed to 
a user. As represented graphically in each of these figures the 

35 selection by the user enables the user to find an image of 
interest to enlarge it, mark it and to print those images which 
are of interest. 

A further useful aspect of the exemplary embodiment of 
the invention is the ability of a user to identify selected 

40 images for downloading from device 330 to another terminal 
which is connected thereto. Such downloading may be 
accomplished in a manner which provides greater assurance 
that the downloaded images are not altered. This is accom- 
plished in the exemplary embodiment using a feature which 

45 is referred to as an image cart. In reviewing images, a user 
is enabled to click on a rectangular icon adjacent to images 
so as to change the color thereof. As represented in FIG. 84 
these rectangular icons change color responsive to selection 
so as to place the images in the image cart. The positioning 

50 of these icons relative to images can be seen for example in 
displayed images represented in FIGS. 62 through 64. When 
scrolling through the images using the control panel 472 the 
user is selectively enabled to click on those images that they 
find of interest for purposes of downloading by changing the 

55 color of the image cart symbol 474 adjacent to the image of 
interest. As explained in FIG. 78 a viewer icon 476 may be 
selected at any time on the control panel to enable a user to 
quickly view those images that they have included in the 
image cart. 

60 A further useful aspect of the exemplary embodiment of 
the present invention is the ability to transfer the images in 
the image cart from the device 330 in a manner that provides 
greater assurance that the images have not been subject to 
tampering. In the exemplary embodiment of the invention a 

65 user is enabled to download images using the image cart 
feature to a terminal However device 330 is programmed so 
as to include in connection with such images a warning to 
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indicate lo the viewer thereof that the image was not secure 
and may be subject to tampering. Given the ability of current 
computer equipment to do image modification and 
manipulation, this feature assures that images which are 
downloaded without security give any user thereof fair 
notice that the image may not be as originally captured. This 
notice is preferably sent with the downloaded image when 
the data corresponding thereto is transferred to the user 
terminal and the image is output on a display thereof. 

The image cart feature however enables the application of 
a digital signature with images downloaded in the image cart 
along with the associated data. This security feature is 
attained by selecting a key icon 477 in the control panel as 
shown in FIG. 78. In response to selection of the key icon 
477 a user downloading images is presented with a screen of 
the type shown in FIG. 85. The screen advises the user that 
the images are being downloaded as a secure file to assure 
integrity. In addition the user is provided with a password 
which must be input to unlock the package of image and 
transaction data which has been secured with the digital 
signature. In the exemplary embodiment, the images are also 
downloaded with an encryption scheme which is integrated 
with the digital signature technique to assure that only the 
authorized user may access such images. Of course it should 
be understood that this technique is exemplary and in other 
embodiments other approaches to encrypting the data as 
well as techniques for reducing the risk that images have not 
been subject to tampering may be used. 

Thus the new transaction record system and method of the 
present invention achieve the above stated objectives, elimi- 
nate difficulties encountered in the use of prior devices and 
systems, solve, problems and attain the desirable results 
described herein. 

In the foregoing description certain terms have been used 
for brevity, clarity and understanding, however no unnec- 
essary limitations are to be implied therefrom because such 
terms are used for descriptive purposes and are intended to 
be broadly construed. Moreover, the descriptions and illus- 
trations herein are by way of examples and the invention is 
not limited to the exact details shown and described. 

In the following claims any feature described as a means 
for performing a function shall be construed as encompass- 
ing any means known to those skilled in the art as being 
capable of performing the recited function and shall not be 
deemed limited to the particular means shown in the fore- 
going description or mere equivalents thereof having 
described the features, discoveries and principles of the 
invention, the manner in which it is constructed and 
operated, and the advantages and useful results attained; the 
new and useful structures, devices, elements, arrangements, 
parts, combinations, systems, equipment, operations, 
methods, processes and relationships are set forth in the 
appended claims. 

We claim: 

1. A method comprising the steps of: 

(a) storing data corresponding to a sequence in a data store 
in operative connection with a computer, wherein the 
sequence includes data representative of at least one 
triggering event and at least one action to be carried out 
responsive to the triggering event; 

(b) sensing for the triggering event through operation of 
the computer, 

(c) carrying out the at least one action in the sequence 
responsive to the triggering event through operation of 
the computer, wherein the at least one action includes 
capturing at least one image from a camera; and 
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(d) storing in the data store responsive to operation of the 
computer, data representative to the triggering event 
and data corresponding to the at least one image. 

2. The method according to claim 1 and prior to step (b) 
s further comprising the steps of periodically capturing 

images from the camera; and storing in the data store in 
correlated relation with the data representative of the trig- 
gering event, data corresponding to at least one image 
captured immediately prior to occurrence of the triggering 
10 event. 

3. The method according to claim 1 wherein in step (a) the 
triggering event in the sequence includes sensing an image 
condition corresponding to lack of usable video from a first 
camera, and an action in the sequence includes capturing an 

!5 image from a second camera, wherein in step (d) data 
corresponding to an image from the second camera is stored 
in the data store. 

4. The method according to claim 3 wherein sensing lack 
of usable video includes sensing lack of contrast in an image 

20 captured from the first camera. 

5. The method according to claim 3 wherein sensing lack 
of usable video includes sensing intensity of a plurality of 
pixels included in the image and determining that a quantity 
of the pixels are either above or below high and low 

25 thresholds, respectively. 

6. The method according to claim 1 wherein in step (a) the 
triggering event includes sensing motion by detecting dif- 
ferences between a plurality of images captured from a first 
camera. 

30 7. The method according to claim 6 and prior to step (b) 
further comprising the step of selecting a detection area 
within a field of view of the first camera, the detection area 
being less than the field of view, and wherein in step (a) the 
triggering event in the sequence includes sensing motion in 

35 the detection area. 

8. The method according to claim 6 and prior to step (b) 
further comprising the step of: 

(e) storing in operative connection with the processor, 
degree data wherein the degree data corresponds to a 

40 degree of change in a plurality of pixels in the detection 
area, which degree of change corresponds to motion, 
and wherein step (b) includes sensing for the degree of 
change in the plurality of pixels in the detection area 
between a plurality of captured images. 
45 9. The method according to claim 8 wherein in step (e) the 
degree data corresponds to both a change in property of the 
pixels in the detection area and a quantity of the pixels which 
undergo such change in property. 

10. The method according to claim 1 wherein at least one 
50 action in the sequence includes data representative of send- 
ing at least one e-mail to an e-mail address, wherein in step 
(c) the computer is operative to cause the e-mail to be sent. 

11. The method according to claim 10 wherein the e-mail 
sent in step (c) includes therewith at least one image 

55 captured by the camera. 

12. The method according to claim 10 and further com- 
prising prior to step (b) storing a group comprising a 
plurality of e-mail addresses in the data store, to be notified 
in the event of the occurrence of a triggering event, and 

60 wherein in step (a) an action in the sequence includes 
sending an e-mail message to the group, and wherein in step 
(c) the computer is operative to cause an e-mail message to 
be sent to the group. 

13. The method according to claim 1 and further com- 
65 prising prior to step (b) the steps of: 

storing in the data store data corresponding to at least one 
recognizable image; 
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and wherein in step (a) the triggering event includes the 
presence of the at least one recognizable image in an 
image captured by the camera. 

14. The method according to claim 13 wherein the rec- 
ognizable image includes at least one human face. 

15. The method according to claim 13 wherein the rec- 
ognizable image includes at least one weapon. 

16. The method according to claim 13 wherein the rec- 
ognizable image includes at least one type of clothing. 

17. The method according to claim 1 and prior to step (a) 
further comprising the step of: 

assigning descriptive designators to each of a plurality of 
cameras, and wherein the sequence data stored in step 
(a) is input using the descriptive designators. 

18. The method according to claim 17 wherein in step (d) 
the data representative to the triggering event includes a 
descriptive designator corresponding to a camera capturing 
the image. 

19. The method according to claim 1 wherein the com- 
puter is operatively connected to an automated banking 
machine and wherein in step (a) at least one triggering event 
in a sequence corresponds to conduct at least one transaction 
step at the automated banking machine, and wherein an 
action in the sequence includes capturing an image of a user 
of the banking machine, and wherein in step (d) an image of 
the user is stored in the data store. 

20. The method according to claim 19 and further com- 
prising the steps of: 

conducting a transaction at the automated banking 
machine, wherein the conduct of the transaction is 
operative to cause the generation of transaction data 
corresponding to the transaction; 

and wherein in step (d) data representative of the trans- 
action data is stored in the data store. 

21. The method according to claim 20 wherein the trans- 
action data stored in (d) includes data representative of at 
least one time associated with the transaction. 

22. The method according to claim 20 wherein the trans- 
action data stored in (d) includes data representative of a 
user name associated with the user conducting the transac- 
tion. 

23. The method according to claim 20 wherein the trans- 
action data stored in step (d) includes data representative of 
an account number associated with the transaction. 

24. The method according to claim 20 wherein the trans- 
action data stored in (d) includes data representative of a 
transaction type associated with the transaction. 

25. The method according to claim 20 wherein the trans- 
action data stored in step (d) includes data representative of 
an amount associated with the transaction. . 

26. The method according to claim 21 and further com- 
prising the steps of searching the data stored in the data store 

. by transaction time, and recovering from the data store at 
least one stored image based on the transaction time asso- 
ciated with the transaction. 

27. The method according to claim 26 wherein the trans- 
action time includes a time period, wherein in the recovery 
step images are recovered responsive to a time associated 
with a transaction being within the time period. 

28. The method according to claim 24 and further com- 
prising the steps of searching the data stored in the data store 
by transaction type, and recovering from the data store at 
least one stored image based on transaction type associated 
with the transaction. 

29. The method according to claim 20 wherein the trans- 
action data includes at least one transaction parameter, 
wherein the transaction parameter includes at least one of a 
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time, a user name, an account number, an amount and a 
transaction type. 

30. The method according to claim 29 and further com- 
prising the steps of searching the data stored in the data store 

5 by at least one transaction parameter associated with 
transactions, and recovering from the data store at least one 
stored image corresponding to the at least one parameter. 

31. The method according to claim 29 wherein the trans- 
action data includes at least two parameters, and further 

10 comprising the steps of searching the transaction data stored 
in the data store for the at least two parameters, and 
recovering from the data store at least one image corre- 
sponding to a transaction having the at least two parameters 
associated therewith. 
15 32. The method according to claim 29 wherein the trans- 
action data includes at least two transaction parameters, and 
further comprising the steps of: 

including an image deletion routine in operative connec- 
tion with the computer operative to delete images 
20 corresponding to a first transaction parameter from the 
data store while retaining images corresponding to a 
second transaction parameter; 
deleting data from the data store corresponding to images 
responsive to operation of the image deletion routine in 
25 the computer, wherein the deleted images correspond to 
the first transaction parameter. 

33. The method according to claim 32 and further com- 
prising the step of sensing for available storage in the data 
store, and wherein in the deleting step the deletion of data is 

30 initiated responsive to available storage reaching a first 
level. 

34. The method according to claim 32 wherein in the 
deleting step the deletion of data is stopped responsive to 
available storage reaching a second level. 

35 35. The method according to claim 1 wherein at least one 
of the actions in the sequence stored in step (a) includes 
capturing images on a generally continuous basis from at 
least one camera, and wherein in step (d) the generally 
continuous video images are stored in the data store. 

40 36. The method according to claim 35 wherein the gen- 
erally continuous video images stored in step (d) are cap- 
tured at a rate of at least ten frames per second. 

37. The method according to claim 1 and further com- 
prising the step of: 

45 selectively deleting from the data store data correspond- 
ing to images by a type of triggering event causing the 
image to be stored, wherein data corresponding to 
images associated with at least one type of triggering 
event is deleted from the data store and wherein data 

50 corresponding to images associated with at least one 
other type of triggering event is retained. 

38. The method according to claim 37 wherein a first type 
of triggering event is associated with an alarm condition, and 
a second type of triggering event is associated with other 

55 than an alarm condition, wherein in the deleting step data 
corresponding to images associated with the second type of 
triggering event are deleted. 

39. The method according to claim 37 and further com- 
prising the step of storing in connection with the computer 

60 an image deletion routine, and wherein the computer oper- 
ates to delete data corresponding to images associated with 
the at least one type of triggering event responsive to storage 
in the data store reaching a first level. 

40. The method according to claim 39 wherein the com- 
65 puter is operative to discontinue the deletion of image data 

responsive to the storage in the data store reaching a second 
level. 
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41. The method according to claim 37 and further com- 
prising storing an image deletion routine in operative con- 
nection with the computer, wherein the image deletion 
routine is operative to cause the computer to execute the 
deleting step wherein data corresponding to images is 
deleted in response to a type of triggering event causing the 
image to be stored and an age of the stored data correspond- 
ing to the image. 

42. The method according to claim 1 wherein the trig- 
gering event corresponds to a time. 

43. The method according to claim 1 and further com- 1 
prising the steps of: 

storing in operative connection with the computer, data 
representative of at least one visual characteristic in the 

searching the data corresponding to images in the data 1 

store for the visual characteristic; 
identifying those images including the visual character- 

44. The method according to claim 43 wherein the visual 
characteristic includes at least one physical characteristic. 

45. The method according to claim 43 wherein the visual 
characteristic includes at least one facial characteristic. 

46. The method according to claim 45 wherein the facial 
characteristic corresponds to a face of a particular person. 

47. The method according to claim 43 wherein the visual 2 
characteristic includes a particular article of apparel. 

48. The method according to claim 43 wherein the visual 
characteristic includes a particular color. 

49. The method according to claim 43 wherein the visual 
characteristic includes an image of a particular device. 

50. The method according to claim 49 wherein the device 
includes a weapon. v - 

51. The method according to claim 1 and further com- 
prising the steps of; 

storing in the data store, data corresponding to a plurality 
of images associated with a plurality of triggering 

displaying at least one image corresponding to each 
triggering event on a display. 4| 

52. The method according to claim 51 wherein actions in 
at least one sequence include capturing and storing data 
corresponding to a plurality of images, and wherein in the 
displaying step a plurality of images corresponding to the 
triggering event are displayed as a set on the display. 4 , 

53. The method according to claim 52 and further com- 
prising the step of capturing arid storing in a data store data 
corresponding to a prior image captured by at least one 
camera prior to occurrence of the triggering event, wherein 
the prior image is included in the displayed set. 5| 

54. The method according to claim 51 and further com- 
prising the steps of: 

providing on the display at least one icon, and navigating 
through a plurality of images responsive to selection of 
the icon with an input device. 5 , 

55. The method according to claim 51 and prior to step (d) 
further comprising the step of storing a data compression 
parameter in operative connection with the triggering event, 
wherein the data corresponding to the image stored in step 
(d) is compressed in accordance with the data compression 6( 
parameter. 

56. The method according to claim 1 and further com- 
prising the steps of: 

providing a server in operative connection with the data 
store; 6i 

accessing stored data corresponding to the image at a user 
terminal in operative connection with the server. 
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57. The method according to claim 56 and further com- 
prising the step of including through operation of the com- 
puter a digital signature with the accessed stored data 
corresponding to the image. 

58. The method according to claim 56 and further com- 
prising the step of transferring data corresponding to the 
image to the user terminal. 

59. The method according to claim 58 wherein the step of 
transferring the image to the user terminal includes trans- 
ferring a key which enables reproducing the image at the 
user terminal. 

60. The method according to claim 56 and further com- 
prising the steps of; 

requesting transfer of the data corresponding to image not 
including a digital signature with the user terminal; 

including through operation of the computer an indicator 
in connection with the data corresponding to the image 
that the image may have been subject to modification; 

transferring the data corresponding to the image with the 
indicator to the user terminal. 

61. The method according to claim 60 and further com- 
prising the steps of: 

providing an input through an input device at the user 
terminal that an image is to include a digital signature; 

including through operation of the computer a digital 
signature in connection with the image; 

transferring the data corresponding to the image to the 
user terminal with a key usable to verify the genuine- 
ness of the image. 

62. The method according to claim 51 and further com- 
prising the steps of: 

selecting at least one image through an input to an input 
device, and transferring the data corresponding to the 
selected image to a remote user terminal. 

63. The method according to claim 62 and wherein the 
input is operative to cause a visible marking indicating the 
selected image to be included on the display. 

64. The method according to claim 62 wherein in the 
selecting step a plurality of images are selected, and further 
comprising transferring the data corresponding to the plu- 
rality of images as a batch to the remote user terminal. 

65. The method according to claim 64 and further com- 
prising including a digital signature in each of the plurality 
of transferred images responsive to operation of the com- 

66. The method according to claim 64 wherein in the 
transferring step the data corresponding to the images is 
transferred to a remote user terminal through a network. 

67. The method according to claim 66 wherein the data 
corresponding to the images is transferred through the 
Internet. 

68. A method comprising: 

(a) storing data corresponding to a sequence in a data store 
in operative connection with a computer, wherein the 
computer is operatively connected to an automated 
banking machine, wherein the sequence includes 
sequence data representative of at least one action to be 
carried out responsive to at least one triggering event, 
wherein the at least one triggering event corresponds to 
conducting at least one transaction step at the auto- 
mated banking machine; 

(b) detecting the least one triggering event through opera- 
tion of the computer; 

(c) carrying out the at least one action in the sequence 
responsive to the triggering event through operation of 
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the computer, wherein the at least one action includes 
capturing at least one image; 
(d) storing in the data store responsive to operation of the 
computer, data related to the triggering event and data 
corresponding to the at least one image. 
69. The method according to claim 68 wherein (c) 
includes capturing at least one image of a user of the 
automated banking machine, and wherein (d) includes stor- 
ing at least one image of the user in the data store. 
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70. The method according to claim 69 and further com- 
prising 

(e) prior to (b), capturing at least one pre-triggering event 
image; and 

(0 storing the at least one pre-triggering event image in 
the data store in correlated relation with the data related 
to the triggering event. 
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